1077 lines
123 KiB
Plaintext
1077 lines
123 KiB
Plaintext
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:50:38 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:50:38 [utils.py:253] non-default args: {'model_tag': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'host': '127.0.0.1', 'model': 'cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', 'trust_remote_code': True, 'max_model_len': 24576, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.95, 'max_num_seqs': 64}
|
|
[0;36m(APIServer pid=43391)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:50:42 [model.py:629] Resolved architecture: Qwen3MoeForCausalLM
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:50:42 [model.py:1755] Using max model len 24576
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:50:42 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(EngineCore_DP0 pid=43554)[0;0m INFO 12-09 20:50:46 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=24576, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
|
[0;36m(EngineCore_DP0 pid=43554)[0;0m WARNING 12-09 20:50:46 [multiproc_executor.py:880] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
INFO 12-09 20:50:49 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:43809 backend=nccl
|
|
INFO 12-09 20:50:49 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:43809 backend=nccl
|
|
INFO 12-09 20:50:49 [pynccl.py:111] vLLM is using nccl==2.27.3
|
|
INFO 12-09 20:50:50 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
|
INFO 12-09 20:50:50 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:50:50 [gpu_model_runner.py:3544] Starting to load model cpatonn/Qwen3-Coder-30B-A3B-Instruct-GPTQ-4bit...
|
|
[0;36m(Worker_TP1 pid=43637)[0;0m INFO 12-09 20:50:50 [compressed_tensors_wNa16.py:114] Using ConchLinearKernel for CompressedTensorsWNA16
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:50:50 [compressed_tensors_wNa16.py:114] Using ConchLinearKernel for CompressedTensorsWNA16
|
|
[0;36m(Worker_TP1 pid=43637)[0;0m INFO 12-09 20:50:50 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(Worker_TP1 pid=43637)[0;0m INFO 12-09 20:50:50 [layer.py:379] Enabled separate cuda stream for MoE shared_experts
|
|
[0;36m(Worker_TP1 pid=43637)[0;0m INFO 12-09 20:50:50 [compressed_tensors_moe.py:189] Using CompressedTensorsWNA16MoEMethod
|
|
[0;36m(Worker_TP1 pid=43637)[0;0m WARNING 12-09 20:50:50 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:50:50 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:50:50 [layer.py:379] Enabled separate cuda stream for MoE shared_experts
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:50:50 [compressed_tensors_moe.py:189] Using CompressedTensorsWNA16MoEMethod
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m WARNING 12-09 20:50:50 [compressed_tensors.py:742] Acceleration for non-quantized schemes is not supported by Compressed Tensors. Falling back to UnquantizedLinearMethod
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 2.20it/s]
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:02<00:02, 1.34s/it]
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:04<00:01, 1.55s/it]
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:06<00:00, 1.75s/it]
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:06<00:00, 1.57s/it]
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:50:57 [default_loader.py:308] Loading weights took 6.31 seconds
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:50:58 [gpu_model_runner.py:3626] Model loading took 8.1992 GiB memory and 7.083759 seconds
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:51:04 [backends.py:616] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/b5f601a270/rank_0_0/backbone for vLLM's torch.compile
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:51:04 [backends.py:676] Dynamo bytecode transform time: 5.57 s
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:51:07 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(Worker_TP1 pid=43637)[0;0m INFO 12-09 20:51:07 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(Worker_TP1 pid=43637)[0;0m WARNING 12-09 20:51:07 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m WARNING 12-09 20:51:07 [fused_moe.py:888] Using default MoE config. Performance might be sub-optimal! Config file not found at ['/opt/venv/lib/python3.13/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=AMD-gfx1201,dtype=int4_w4a16.json']
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:51:09 [backends.py:260] Compiling a graph for compile range (1, 2048) takes 2.69 s
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:51:09 [monitor.py:34] torch.compile takes 8.26 s in total
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:51:12 [gpu_worker.py:364] Available KV cache memory: 19.36 GiB
|
|
[0;36m(EngineCore_DP0 pid=43554)[0;0m INFO 12-09 20:51:12 [kv_cache_utils.py:1287] GPU KV cache size: 422,960 tokens
|
|
[0;36m(EngineCore_DP0 pid=43554)[0;0m INFO 12-09 20:51:12 [kv_cache_utils.py:1292] Maximum concurrency for 24,576 tokens per request: 17.21x
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:07, 2.41it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:06, 2.44it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:01<00:06, 2.45it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:01<00:06, 2.46it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:02<00:05, 2.47it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:02<00:05, 2.47it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:02<00:04, 2.47it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:03<00:04, 2.48it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:03<00:04, 2.47it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:04<00:03, 2.46it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:04<00:03, 2.45it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:04<00:02, 2.44it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:05<00:02, 2.45it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:05<00:02, 2.44it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:06<00:01, 2.44it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:06<00:01, 2.44it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:06<00:00, 2.43it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:07<00:00, 2.42it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 2.42it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 2.44it/s]
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (decode, FULL): 9%|▉ | 1/11 [00:00<00:04, 2.48it/s]
|
|
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:03, 2.49it/s]
|
|
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:01<00:03, 2.50it/s]
|
|
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:01<00:02, 2.50it/s]
|
|
Capturing CUDA graphs (decode, FULL): 45%|████▌ | 5/11 [00:01<00:02, 2.51it/s]
|
|
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:02<00:01, 2.52it/s]
|
|
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:02<00:01, 2.52it/s]
|
|
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:03<00:01, 2.51it/s]
|
|
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:03<00:00, 2.51it/s]
|
|
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:03<00:00, 2.52it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:04<00:00, 2.50it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:04<00:00, 2.51it/s]
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:51:25 [gpu_model_runner.py:4548] Graph capturing finished in 13 secs, took 1.58 GiB
|
|
[0;36m(EngineCore_DP0 pid=43554)[0;0m INFO 12-09 20:51:25 [core.py:256] init engine (profile, create kv cache, warmup model) took 27.07 seconds
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [api_server.py:1099] Supported tasks: ['generate']
|
|
[0;36m(APIServer pid=43391)[0;0m WARNING 12-09 20:51:27 [model.py:1581] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [serving_responses.py:197] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [serving_completion.py:73] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [serving_chat.py:133] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:38] Available routes are:
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /docs, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /tokenize, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /detokenize, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /pause, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /resume, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /is_paused, Methods: GET
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /metrics, Methods: GET
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /health, Methods: GET
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /load, Methods: GET
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/models, Methods: GET
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /version, Methods: GET
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/responses, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/messages, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/completions, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /ping, Methods: GET
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /ping, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /invocations, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /classify, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /score, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/score, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /rerank, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:27 [launcher.py:46] Route: /pooling, Methods: POST
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: Started server process [43391]
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: Waiting for application startup.
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: Application startup complete.
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55700 - "GET /v1/models HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44228 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44238 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:47 [loggers.py:248] Engine 000: Avg prompt throughput: 84.0 tokens/s, Avg generation throughput: 84.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:51:57 [loggers.py:248] Engine 000: Avg prompt throughput: 136.8 tokens/s, Avg generation throughput: 160.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34916 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:52:07 [loggers.py:248] Engine 000: Avg prompt throughput: 207.4 tokens/s, Avg generation throughput: 203.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:52:17 [loggers.py:248] Engine 000: Avg prompt throughput: 224.2 tokens/s, Avg generation throughput: 157.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:52:27 [loggers.py:248] Engine 000: Avg prompt throughput: 277.0 tokens/s, Avg generation throughput: 187.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:52:37 [loggers.py:248] Engine 000: Avg prompt throughput: 310.2 tokens/s, Avg generation throughput: 220.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40560 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40530 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:51904 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:52:47 [loggers.py:248] Engine 000: Avg prompt throughput: 452.9 tokens/s, Avg generation throughput: 209.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:34918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:52:57 [loggers.py:248] Engine 000: Avg prompt throughput: 185.7 tokens/s, Avg generation throughput: 190.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:53:07 [loggers.py:248] Engine 000: Avg prompt throughput: 299.2 tokens/s, Avg generation throughput: 231.9 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55352 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55482 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55512 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:53:17 [loggers.py:248] Engine 000: Avg prompt throughput: 310.1 tokens/s, Avg generation throughput: 334.7 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:49820 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55496 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:53:27 [loggers.py:248] Engine 000: Avg prompt throughput: 132.9 tokens/s, Avg generation throughput: 250.4 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:53:37 [loggers.py:248] Engine 000: Avg prompt throughput: 109.9 tokens/s, Avg generation throughput: 201.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:53:47 [loggers.py:248] Engine 000: Avg prompt throughput: 246.6 tokens/s, Avg generation throughput: 207.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:43318 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:53:57 [loggers.py:248] Engine 000: Avg prompt throughput: 184.0 tokens/s, Avg generation throughput: 258.4 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55368 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55484 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:40546 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55354 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:54:07 [loggers.py:248] Engine 000: Avg prompt throughput: 130.1 tokens/s, Avg generation throughput: 253.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59300 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59312 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59322 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:54:17 [loggers.py:248] Engine 000: Avg prompt throughput: 80.4 tokens/s, Avg generation throughput: 80.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59312 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59300 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59350 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59312 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59336 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:54:27 [loggers.py:248] Engine 000: Avg prompt throughput: 140.2 tokens/s, Avg generation throughput: 222.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59350 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59322 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59350 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59322 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:58252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:58260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59300 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:58260 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:58274 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:58286 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:58286 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:55384 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:54:37 [loggers.py:248] Engine 000: Avg prompt throughput: 275.0 tokens/s, Avg generation throughput: 212.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59350 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:59324 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:54:47 [loggers.py:248] Engine 000: Avg prompt throughput: 50.4 tokens/s, Avg generation throughput: 220.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:54:57 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 74.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:55:07 [loggers.py:248] Engine 000: Avg prompt throughput: 434.0 tokens/s, Avg generation throughput: 201.0 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 9.7%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53548 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:55:17 [loggers.py:248] Engine 000: Avg prompt throughput: 1294.3 tokens/s, Avg generation throughput: 581.5 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 29.9%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:55:27 [loggers.py:248] Engine 000: Avg prompt throughput: 892.1 tokens/s, Avg generation throughput: 758.8 tokens/s, Running: 34 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 39.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47794 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53548 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:55:37 [loggers.py:248] Engine 000: Avg prompt throughput: 670.6 tokens/s, Avg generation throughput: 868.0 tokens/s, Running: 44 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 44.4%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47794 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53548 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:55:47 [loggers.py:248] Engine 000: Avg prompt throughput: 671.7 tokens/s, Avg generation throughput: 842.1 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 47.2%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47794 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53548 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:56532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47794 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:55:57 [loggers.py:248] Engine 000: Avg prompt throughput: 943.0 tokens/s, Avg generation throughput: 802.8 tokens/s, Running: 43 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 42.1%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38350 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38376 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47794 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:56532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:56532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:56532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53548 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47794 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:56:07 [loggers.py:248] Engine 000: Avg prompt throughput: 1003.2 tokens/s, Avg generation throughput: 897.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 37.8%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53548 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:56:17 [loggers.py:248] Engine 000: Avg prompt throughput: 434.6 tokens/s, Avg generation throughput: 840.7 tokens/s, Running: 41 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 36.2%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47746 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38350 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53548 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38376 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:56532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35292 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35306 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35308 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35308 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53038 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35308 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:56:27 [loggers.py:248] Engine 000: Avg prompt throughput: 968.1 tokens/s, Avg generation throughput: 976.4 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 33.0%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44906 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47794 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38350 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:56532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47794 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:56:37 [loggers.py:248] Engine 000: Avg prompt throughput: 629.3 tokens/s, Avg generation throughput: 1008.7 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 31.3%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44946 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53540 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38410 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38376 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53562 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44986 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:56532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45012 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47802 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:56:47 [loggers.py:248] Engine 000: Avg prompt throughput: 1077.6 tokens/s, Avg generation throughput: 833.7 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 29.2%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53548 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44938 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:56:57 [loggers.py:248] Engine 000: Avg prompt throughput: 635.2 tokens/s, Avg generation throughput: 744.8 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 27.8%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35308 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:57:07 [loggers.py:248] Engine 000: Avg prompt throughput: 766.8 tokens/s, Avg generation throughput: 721.7 tokens/s, Running: 33 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 26.3%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35308 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:57:17 [loggers.py:248] Engine 000: Avg prompt throughput: 661.1 tokens/s, Avg generation throughput: 823.3 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 25.3%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45078 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:56532 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42396 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35308 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:57:27 [loggers.py:248] Engine 000: Avg prompt throughput: 903.0 tokens/s, Avg generation throughput: 695.2 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 23.9%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:35308 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47740 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:57:37 [loggers.py:248] Engine 000: Avg prompt throughput: 759.7 tokens/s, Avg generation throughput: 740.9 tokens/s, Running: 33 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 22.8%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38424 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44996 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53058 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38366 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:57:47 [loggers.py:248] Engine 000: Avg prompt throughput: 533.6 tokens/s, Avg generation throughput: 815.8 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 22.1%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45030 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44990 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53046 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44954 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47808 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53574 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45056 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44138 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:47730 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53040 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:57:57 [loggers.py:248] Engine 000: Avg prompt throughput: 1133.1 tokens/s, Avg generation throughput: 780.8 tokens/s, Running: 41 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 20.7%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53594 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53066 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45076 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:45022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38364 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53176 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53196 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:38392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:42392 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:44982 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53216 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53232 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53232 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53234 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53238 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: 127.0.0.1:53246 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:58:07 [loggers.py:248] Engine 000: Avg prompt throughput: 259.0 tokens/s, Avg generation throughput: 882.2 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 20.4%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:58:17 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 411.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 20.4%
|
|
[0;36m(APIServer pid=43391)[0;0m INFO 12-09 20:58:24 [launcher.py:110] Shutting down FastAPI HTTP server.
|
|
[0;36m(Worker_TP0 pid=43636)[0;0m INFO 12-09 20:58:24 [multiproc_executor.py:709] Parent process exited, terminating worker
|
|
[0;36m(Worker_TP1 pid=43637)[0;0m INFO 12-09 20:58:24 [multiproc_executor.py:709] Parent process exited, terminating worker
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: Shutting down
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: Waiting for application shutdown.
|
|
[0;36m(APIServer pid=43391)[0;0m INFO: Application shutdown complete.
|