benchmarks/benchmark_results_nvidia-ada5000/cpatonn_Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit_tp1_server.log

WARNING 12-10 20:10:10 [argparse_utils.py:195] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in v0.13.
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:10 [api_server.py:1772] vLLM API server version 0.12.0
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:10 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'trust_remote_code': True, 'max_model_len': 24576, 'max_num_seqs': 64}
[0;36m(APIServer pid=5918)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:17 [model.py:637] Resolved architecture: Qwen3MoeForCausalLM
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:17 [model.py:1750] Using max model len 24576
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:18 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:23 [core.py:93] Initializing a V1 LLM engine (v0.12.0) with config: model='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=24576, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01), seed=0, served_model_name=cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>}, 'local_cache_dir': None}
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:23 [parallel_state.py:1200] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.3:55455 backend=nccl
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:23 [parallel_state.py:1408] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:24 [gpu_model_runner.py:3467] Starting to load model cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit...
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:24 [compressed_tensors_wNa16.py:114] Using MarlinLinearKernel for CompressedTensorsWNA16
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:24 [cuda.py:411] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:24 [layer.py:379] Enabled separate cuda stream for MoE shared_experts
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:24 [compressed_tensors_moe.py:167] Using CompressedTensorsWNA16MarlinMoEMethod
[0;36m(EngineCore_DP0 pid=6022)[0;0m WARNING 12-10 20:10:24 [compressed_tensors.py:721] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:01,  1.85it/s]
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:02<00:03,  1.58s/it]
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:04<00:01,  1.83s/it]
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:07<00:00,  2.06s/it]
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:07<00:00,  1.84s/it]
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:32 [default_loader.py:308] Loading weights took 7.48 seconds
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:33 [gpu_model_runner.py:3549] Model loading took 15.6116 GiB memory and 9.101340 seconds
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:39 [backends.py:655] Using cache directory: /root/.cache/vllm/torch_compile_cache/de3f9d1fa4/rank_0_0/backbone for vLLM's torch.compile
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:39 [backends.py:715] Dynamo bytecode transform time: 5.66 s
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:40 [backends.py:257] Cache the graph for dynamic shape for later use
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:47 [backends.py:288] Compiling a graph for dynamic shape takes 7.49 s
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:49 [monitor.py:34] torch.compile takes 13.15 s in total
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:49 [gpu_worker.py:359] Available KV cache memory: 12.20 GiB
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:50 [kv_cache_utils.py:1286] GPU KV cache size: 133,216 tokens
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:50 [kv_cache_utils.py:1291] Maximum concurrency for 24,576 tokens per request: 5.42x
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  16%|█▌        | 3/19 [00:00<00:00, 21.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  32%|███▏      | 6/19 [00:00<00:00, 23.30it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  47%|████▋     | 9/19 [00:00<00:00, 24.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  63%|██████▎   | 12/19 [00:00<00:00, 24.52it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  79%|███████▉  | 15/19 [00:00<00:00, 24.85it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  95%|█████████▍| 18/19 [00:00<00:00, 25.21it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:00<00:00, 24.46it/s]
[0;36m(EngineCore_DP0 pid=6022)[0;0m 
Capturing CUDA graphs (decode, FULL):   0%|          | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):  27%|██▋       | 3/11 [00:00<00:00, 23.62it/s]
Capturing CUDA graphs (decode, FULL):  55%|█████▍    | 6/11 [00:00<00:00, 25.30it/s]
Capturing CUDA graphs (decode, FULL):  82%|████████▏ | 9/11 [00:00<00:00, 25.99it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 25.86it/s]
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:52 [gpu_model_runner.py:4466] Graph capturing finished in 2 secs, took 0.37 GiB
[0;36m(EngineCore_DP0 pid=6022)[0;0m INFO 12-10 20:10:52 [core.py:254] init engine (profile, create kv cache, warmup model) took 18.43 seconds
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:53 [api_server.py:1520] Supported tasks: ['generate']
[0;36m(APIServer pid=5918)[0;0m WARNING 12-10 20:10:53 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:53 [serving_responses.py:194] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:53 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:53 [serving_completion.py:73] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [api_server.py:1847] Starting vLLM API server 0 on http://127.0.0.1:8000
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:38] Available routes are:
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /docs, Methods: HEAD, GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /health, Methods: GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /load, Methods: GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /pause, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /resume, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /is_paused, Methods: GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /tokenize, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /detokenize, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/models, Methods: GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /version, Methods: GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/responses, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/messages, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/completions, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /ping, Methods: GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /ping, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /invocations, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /metrics, Methods: GET
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /classify, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/embeddings, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /score, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/score, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /rerank, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v1/rerank, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /v2/rerank, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:10:54 [launcher.py:46] Route: /pooling, Methods: POST
[0;36m(APIServer pid=5918)[0;0m INFO:     Started server process [5918]
[0;36m(APIServer pid=5918)[0;0m INFO:     Waiting for application startup.
[0;36m(APIServer pid=5918)[0;0m INFO:     Application startup complete.
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:54364 - "GET /v1/models HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34322 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34332 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34332 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:11:14 [loggers.py:236] Engine 000: Avg prompt throughput: 84.0 tokens/s, Avg generation throughput: 135.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34322 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34322 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34322 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:11:24 [loggers.py:236] Engine 000: Avg prompt throughput: 136.8 tokens/s, Avg generation throughput: 186.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34322 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41568 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34322 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:11:34 [loggers.py:236] Engine 000: Avg prompt throughput: 207.4 tokens/s, Avg generation throughput: 172.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:46224 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:46224 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:46224 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:11:44 [loggers.py:236] Engine 000: Avg prompt throughput: 224.2 tokens/s, Avg generation throughput: 170.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:46224 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38026 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38026 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:11:54 [loggers.py:236] Engine 000: Avg prompt throughput: 334.8 tokens/s, Avg generation throughput: 191.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:46224 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38026 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50386 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38026 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50386 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:12:04 [loggers.py:236] Engine 000: Avg prompt throughput: 252.4 tokens/s, Avg generation throughput: 214.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:38034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:46224 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50386 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:46224 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50386 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36802 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34310 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36816 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36816 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:12:14 [loggers.py:236] Engine 000: Avg prompt throughput: 452.8 tokens/s, Avg generation throughput: 290.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50386 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50386 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:12:24 [loggers.py:236] Engine 000: Avg prompt throughput: 185.7 tokens/s, Avg generation throughput: 87.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49000 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49000 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49018 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49032 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49018 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49032 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49032 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49000 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:12:34 [loggers.py:236] Engine 000: Avg prompt throughput: 299.2 tokens/s, Avg generation throughput: 383.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50386 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49000 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49018 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49032 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49000 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:12:44 [loggers.py:236] Engine 000: Avg prompt throughput: 310.1 tokens/s, Avg generation throughput: 369.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50386 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49018 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49018 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:12:54 [loggers.py:236] Engine 000: Avg prompt throughput: 132.9 tokens/s, Avg generation throughput: 177.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49018 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:13:04 [loggers.py:236] Engine 000: Avg prompt throughput: 109.9 tokens/s, Avg generation throughput: 201.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41136 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:13:14 [loggers.py:236] Engine 000: Avg prompt throughput: 246.6 tokens/s, Avg generation throughput: 194.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41136 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41148 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41150 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41148 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41148 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41136 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41148 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41136 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:13:24 [loggers.py:236] Engine 000: Avg prompt throughput: 184.0 tokens/s, Avg generation throughput: 331.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:48996 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41136 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41148 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:36810 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49034 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:41148 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:13:34 [loggers.py:236] Engine 000: Avg prompt throughput: 130.1 tokens/s, Avg generation throughput: 128.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50492 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50492 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45656 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45660 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45664 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:13:44 [loggers.py:236] Engine 000: Avg prompt throughput: 80.4 tokens/s, Avg generation throughput: 103.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45664 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45660 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50492 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45664 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50492 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45660 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45660 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:13:54 [loggers.py:236] Engine 000: Avg prompt throughput: 140.2 tokens/s, Avg generation throughput: 273.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45664 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50492 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45660 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45664 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50492 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45660 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45660 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:51162 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:51174 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:51180 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50492 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45664 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:14:04 [loggers.py:236] Engine 000: Avg prompt throughput: 275.0 tokens/s, Avg generation throughput: 179.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:50492 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:45664 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:51162 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:14:14 [loggers.py:236] Engine 000: Avg prompt throughput: 50.4 tokens/s, Avg generation throughput: 228.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:14:24 [loggers.py:236] Engine 000: Avg prompt throughput: 138.6 tokens/s, Avg generation throughput: 107.7 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 3.2%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32814 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32814 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:14:34 [loggers.py:236] Engine 000: Avg prompt throughput: 1095.6 tokens/s, Avg generation throughput: 581.2 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 23.4%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34668 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32814 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:14:44 [loggers.py:236] Engine 000: Avg prompt throughput: 1092.7 tokens/s, Avg generation throughput: 833.7 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 36.3%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34642 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34668 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:14:54 [loggers.py:236] Engine 000: Avg prompt throughput: 771.5 tokens/s, Avg generation throughput: 977.3 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.5%, Prefix cache hit rate: 42.9%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32862 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32814 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:15:04 [loggers.py:236] Engine 000: Avg prompt throughput: 475.2 tokens/s, Avg generation throughput: 823.5 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 46.3%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32814 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32814 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49032 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49032 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:15:14 [loggers.py:236] Engine 000: Avg prompt throughput: 967.8 tokens/s, Avg generation throughput: 894.4 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.5%, Prefix cache hit rate: 43.9%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49032 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:15:24 [loggers.py:236] Engine 000: Avg prompt throughput: 1023.4 tokens/s, Avg generation throughput: 904.3 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 39.2%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32868 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:15:34 [loggers.py:236] Engine 000: Avg prompt throughput: 585.7 tokens/s, Avg generation throughput: 805.6 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 36.9%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43754 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43760 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53776 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43760 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:15:44 [loggers.py:236] Engine 000: Avg prompt throughput: 887.3 tokens/s, Avg generation throughput: 923.7 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 33.9%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43760 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43756 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:39524 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:15:54 [loggers.py:236] Engine 000: Avg prompt throughput: 647.9 tokens/s, Avg generation throughput: 1108.8 tokens/s, Running: 29 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.6%, Prefix cache hit rate: 31.9%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43754 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34626 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43760 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49012 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32878 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43754 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43760 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:16:04 [loggers.py:236] Engine 000: Avg prompt throughput: 882.7 tokens/s, Avg generation throughput: 812.7 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 29.7%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32876 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53732 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32834 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:16:14 [loggers.py:236] Engine 000: Avg prompt throughput: 916.1 tokens/s, Avg generation throughput: 707.2 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 27.6%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:16:24 [loggers.py:236] Engine 000: Avg prompt throughput: 548.9 tokens/s, Avg generation throughput: 711.5 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 26.5%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:16:34 [loggers.py:236] Engine 000: Avg prompt throughput: 779.2 tokens/s, Avg generation throughput: 783.6 tokens/s, Running: 23 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 25.3%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:16:44 [loggers.py:236] Engine 000: Avg prompt throughput: 963.4 tokens/s, Avg generation throughput: 735.1 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 23.7%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:16:54 [loggers.py:236] Engine 000: Avg prompt throughput: 541.9 tokens/s, Avg generation throughput: 673.6 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 22.9%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32848 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:55416 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:17:04 [loggers.py:236] Engine 000: Avg prompt throughput: 714.3 tokens/s, Avg generation throughput: 797.7 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.2%, Prefix cache hit rate: 21.9%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:55426 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34990 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34678 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53748 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32800 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32778 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34654 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:55426 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34984 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49016 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO 12-10 20:17:14 [loggers.py:236] Engine 000: Avg prompt throughput: 944.4 tokens/s, Avg generation throughput: 810.2 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 20.8%
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43812 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34662 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:49022 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:55416 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53774 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:55416 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34666 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:34628 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32806 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32840 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35004 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53780 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:32832 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43762 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:35006 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:43790 - "POST /v1/completions HTTP/1.1" 200 OK
[0;36m(APIServer pid=5918)[0;0m INFO:     127.0.0.1:53788 - "POST /v1/completions HTTP/1.1" 200 OK