1051 lines
122 KiB
Plaintext
1051 lines
122 KiB
Plaintext
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:10:31 [api_server.py:1351] vLLM API server version 0.11.2.dev690+g67475a6e8.d20251209
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:10:31 [utils.py:253] non-default args: {'model_tag': 'openai/gpt-oss-20b', 'host': '127.0.0.1', 'model': 'openai/gpt-oss-20b', 'trust_remote_code': True, 'max_model_len': 32768, 'gpu_memory_utilization': 0.98, 'max_num_seqs': 64}
|
|
[0;36m(APIServer pid=9049)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:10:35 [model.py:629] Resolved architecture: GptOssForCausalLM
|
|
[0;36m(APIServer pid=9049)[0;0m
|
|
Parse safetensors files: 0%| | 0/3 [00:00<?, ?it/s]
|
|
Parse safetensors files: 33%|███▎ | 1/3 [00:05<00:10, 5.20s/it]
|
|
Parse safetensors files: 100%|██████████| 3/3 [00:05<00:00, 1.73s/it]
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:10:41 [model.py:1755] Using max model len 32768
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:10:41 [scheduler.py:228] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:10:41 [config.py:269] Overriding max cuda graph capture size to 1024 for performance.
|
|
/opt/venv/lib64/python3.13/site-packages/torch/library.py:357: UserWarning: Warning only once for all operators, other operators may also be overridden.
|
|
Overriding a previously registered kernel for the same operator and the same dispatch key
|
|
operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
|
|
registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926
|
|
dispatch key: ADInplaceOrView
|
|
previous kernel: no debug info
|
|
new kernel: registered at /opt/venv/lib64/python3.13/site-packages/torch/_library/custom_ops.py:926 (Triggered internally at /__w/TheRock/TheRock/external-builds/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
|
|
self.m.impl(
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:45 [core.py:93] Initializing a V1 LLM engine (v0.11.2.dev690+g67475a6e8.d20251209) with config: model='openai/gpt-oss-20b', speculative_config=None, tokenizer='openai/gpt-oss-20b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=mxfp4, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='openai_gptoss', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=openai/gpt-oss-20b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 528, 544, 560, 576, 592, 608, 624, 640, 656, 672, 688, 704, 720, 736, 752, 768, 784, 800, 816, 832, 848, 864, 880, 896, 912, 928, 944, 960, 976, 992, 1008, 1024], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 1024, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:45 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.122:37735 backend=nccl
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:45 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:45 [gpu_model_runner.py:3544] Starting to load model openai/gpt-oss-20b...
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:45 [rocm.py:320] Using Triton Attention backend on V1 engine.
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:45 [layer.py:379] Enabled separate cuda stream for MoE shared_experts
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:45 [mxfp4.py:171] Using Triton backend
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m
|
|
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s]
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m
|
|
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:00<00:01, 1.38it/s]
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m
|
|
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:01<00:00, 1.13it/s]
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00, 1.07it/s]
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m
|
|
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00, 1.10it/s]
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:49 [default_loader.py:308] Loading weights took 2.79 seconds
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:49 [gpu_model_runner.py:3626] Model loading took 14.3066 GiB memory and 3.481873 seconds
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:51 [backends.py:616] Using cache directory: /home/kyuz0/.cache/vllm/torch_compile_cache/fd3d592b37/rank_0_0/backbone for vLLM's torch.compile
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:51 [backends.py:676] Dynamo bytecode transform time: 1.70 s
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:10:54 [backends.py:243] Cache the graph of compile range (1, 2048) for later use
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:11:13 [backends.py:260] Compiling a graph for compile range (1, 2048) takes 20.60 s
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:11:13 [monitor.py:34] torch.compile takes 22.30 s in total
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:11:15 [gpu_worker.py:364] Available KV cache memory: 13.34 GiB
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:11:15 [kv_cache_utils.py:1287] GPU KV cache size: 291,392 tokens
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:11:15 [kv_cache_utils.py:1292] Maximum concurrency for 32,768 tokens per request: 16.67x
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/83 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 1%| | 1/83 [00:00<00:14, 5.68it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 2%|▏ | 2/83 [00:00<00:13, 6.10it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 4%|▎ | 3/83 [00:00<00:13, 6.15it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▍ | 4/83 [00:00<00:12, 6.30it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 6%|▌ | 5/83 [00:00<00:12, 6.38it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 7%|▋ | 6/83 [00:00<00:11, 6.51it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 8%|▊ | 7/83 [00:01<00:11, 6.53it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 10%|▉ | 8/83 [00:01<00:11, 6.62it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 9/83 [00:01<00:11, 6.65it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 12%|█▏ | 10/83 [00:01<00:10, 6.80it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 13%|█▎ | 11/83 [00:01<00:10, 6.84it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 14%|█▍ | 12/83 [00:01<00:10, 6.95it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 13/83 [00:01<00:09, 7.03it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 17%|█▋ | 14/83 [00:02<00:09, 7.17it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 18%|█▊ | 15/83 [00:02<00:09, 7.18it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 19%|█▉ | 16/83 [00:02<00:09, 7.19it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|██ | 17/83 [00:02<00:09, 7.27it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 22%|██▏ | 18/83 [00:02<00:08, 7.44it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 23%|██▎ | 19/83 [00:02<00:08, 7.48it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 24%|██▍ | 20/83 [00:02<00:08, 7.62it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 25%|██▌ | 21/83 [00:03<00:08, 7.71it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 22/83 [00:03<00:07, 7.88it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 28%|██▊ | 23/83 [00:03<00:07, 7.88it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 29%|██▉ | 24/83 [00:03<00:07, 8.02it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 30%|███ | 25/83 [00:03<00:07, 8.18it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 31%|███▏ | 26/83 [00:03<00:06, 8.45it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 33%|███▎ | 27/83 [00:03<00:06, 8.50it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 34%|███▎ | 28/83 [00:03<00:06, 8.71it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 35%|███▍ | 29/83 [00:03<00:06, 8.93it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 31/83 [00:04<00:05, 9.31it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 40%|███▉ | 33/83 [00:04<00:05, 9.63it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 35/83 [00:04<00:04, 9.96it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 45%|████▍ | 37/83 [00:04<00:04, 10.28it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 39/83 [00:04<00:04, 10.56it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 49%|████▉ | 41/83 [00:05<00:03, 10.91it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 52%|█████▏ | 43/83 [00:05<00:03, 11.35it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 54%|█████▍ | 45/83 [00:05<00:03, 11.81it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 57%|█████▋ | 47/83 [00:05<00:02, 12.49it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 59%|█████▉ | 49/83 [00:05<00:02, 13.13it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 61%|██████▏ | 51/83 [00:05<00:02, 13.80it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 64%|██████▍ | 53/83 [00:05<00:02, 14.30it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 66%|██████▋ | 55/83 [00:06<00:01, 14.64it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 69%|██████▊ | 57/83 [00:06<00:01, 15.06it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████ | 59/83 [00:06<00:01, 15.70it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 73%|███████▎ | 61/83 [00:06<00:01, 16.15it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 76%|███████▌ | 63/83 [00:06<00:01, 16.73it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 78%|███████▊ | 65/83 [00:06<00:01, 17.26it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 68/83 [00:06<00:00, 18.74it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▌ | 71/83 [00:06<00:00, 19.72it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 74/83 [00:06<00:00, 20.98it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 93%|█████████▎| 77/83 [00:07<00:00, 22.31it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 96%|█████████▋| 80/83 [00:07<00:00, 23.69it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 83/83 [00:07<00:00, 18.88it/s]
|
|
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 83/83 [00:07<00:00, 11.13it/s]
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m
|
|
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
|
|
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:00<00:00, 24.74it/s]
|
|
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:00<00:00, 26.84it/s]
|
|
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:00<00:00, 23.05it/s]
|
|
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 21.40it/s]
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:11:24 [gpu_model_runner.py:4548] Graph capturing finished in 9 secs, took 0.77 GiB
|
|
[0;36m(EngineCore_DP0 pid=9216)[0;0m INFO 12-09 19:11:24 [core.py:256] init engine (profile, create kv cache, warmup model) took 34.56 seconds
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:25 [api_server.py:1099] Supported tasks: ['generate']
|
|
[0;36m(APIServer pid=9049)[0;0m WARNING 12-09 19:11:25 [serving_responses.py:218] For gpt-oss, we ignore --enable-auto-tool-choice and always enable tool use.
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [api_server.py:1425] Starting vLLM API server 0 on http://127.0.0.1:8000
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:38] Available routes are:
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /docs, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /tokenize, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /detokenize, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /pause, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /resume, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /is_paused, Methods: GET
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /metrics, Methods: GET
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /health, Methods: GET
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /load, Methods: GET
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/models, Methods: GET
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /version, Methods: GET
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/responses, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/messages, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/completions, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /ping, Methods: GET
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /ping, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /invocations, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /classify, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/embeddings, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /score, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/score, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /rerank, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v1/rerank, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /v2/rerank, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:28 [launcher.py:46] Route: /pooling, Methods: POST
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: Started server process [9049]
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: Waiting for application startup.
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: Application startup complete.
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56250 - "GET /v1/models HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51756 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:48 [loggers.py:248] Engine 000: Avg prompt throughput: 4.9 tokens/s, Avg generation throughput: 15.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51780 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51780 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:11:58 [loggers.py:248] Engine 000: Avg prompt throughput: 132.7 tokens/s, Avg generation throughput: 85.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:12:08 [loggers.py:248] Engine 000: Avg prompt throughput: 223.5 tokens/s, Avg generation throughput: 139.8 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54306 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43272 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:12:18 [loggers.py:248] Engine 000: Avg prompt throughput: 275.0 tokens/s, Avg generation throughput: 164.6 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43272 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51780 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43272 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:12:28 [loggers.py:248] Engine 000: Avg prompt throughput: 225.3 tokens/s, Avg generation throughput: 184.5 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46534 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51756 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43272 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46394 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:12:38 [loggers.py:248] Engine 000: Avg prompt throughput: 365.1 tokens/s, Avg generation throughput: 200.4 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54306 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51756 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56430 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56438 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:12:48 [loggers.py:248] Engine 000: Avg prompt throughput: 331.9 tokens/s, Avg generation throughput: 217.3 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51756 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51756 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46534 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46378 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:12:58 [loggers.py:248] Engine 000: Avg prompt throughput: 274.5 tokens/s, Avg generation throughput: 212.4 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56430 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51756 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56430 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:13:08 [loggers.py:248] Engine 000: Avg prompt throughput: 288.6 tokens/s, Avg generation throughput: 195.0 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43282 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:13:18 [loggers.py:248] Engine 000: Avg prompt throughput: 174.4 tokens/s, Avg generation throughput: 249.5 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56430 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54306 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56430 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54306 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:13:28 [loggers.py:248] Engine 000: Avg prompt throughput: 377.1 tokens/s, Avg generation throughput: 254.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46726 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:60064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:13:39 [loggers.py:248] Engine 000: Avg prompt throughput: 42.4 tokens/s, Avg generation throughput: 269.5 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51770 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46750 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:13:49 [loggers.py:248] Engine 000: Avg prompt throughput: 196.8 tokens/s, Avg generation throughput: 227.6 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56414 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53418 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:60064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:60064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:13:59 [loggers.py:248] Engine 000: Avg prompt throughput: 243.4 tokens/s, Avg generation throughput: 250.1 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35786 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35772 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:14:09 [loggers.py:248] Engine 000: Avg prompt throughput: 155.6 tokens/s, Avg generation throughput: 231.5 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:14:19 [loggers.py:248] Engine 000: Avg prompt throughput: 63.4 tokens/s, Avg generation throughput: 179.1 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:46742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35194 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36130 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36132 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:14:29 [loggers.py:248] Engine 000: Avg prompt throughput: 156.7 tokens/s, Avg generation throughput: 156.8 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:14:39 [loggers.py:248] Engine 000: Avg prompt throughput: 89.6 tokens/s, Avg generation throughput: 194.1 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43244 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:60064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:35208 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51792 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:43296 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:60064 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34320 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:34320 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:36142 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:14:49 [loggers.py:248] Engine 000: Avg prompt throughput: 256.0 tokens/s, Avg generation throughput: 211.3 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:14:59 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 185.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:15:09 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 67.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:15:19 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 25.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:15:29 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55976 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:15:39 [loggers.py:248] Engine 000: Avg prompt throughput: 214.7 tokens/s, Avg generation throughput: 55.7 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 4.9%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55608 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55660 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:15:49 [loggers.py:248] Engine 000: Avg prompt throughput: 1016.7 tokens/s, Avg generation throughput: 319.2 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 23.2%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55766 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47634 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47692 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55608 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:15:59 [loggers.py:248] Engine 000: Avg prompt throughput: 1218.5 tokens/s, Avg generation throughput: 594.9 tokens/s, Running: 54 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 37.3%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55608 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55660 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55766 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55976 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:16:09 [loggers.py:248] Engine 000: Avg prompt throughput: 707.5 tokens/s, Avg generation throughput: 679.7 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 43.1%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55766 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55608 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:16:19 [loggers.py:248] Engine 000: Avg prompt throughput: 521.6 tokens/s, Avg generation throughput: 778.2 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 46.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47634 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:16:29 [loggers.py:248] Engine 000: Avg prompt throughput: 730.5 tokens/s, Avg generation throughput: 773.8 tokens/s, Running: 64 reqs, Waiting: 13 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 44.9%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39166 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47692 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55660 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55766 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39166 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47634 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:16:39 [loggers.py:248] Engine 000: Avg prompt throughput: 910.8 tokens/s, Avg generation throughput: 768.0 tokens/s, Running: 63 reqs, Waiting: 10 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 40.4%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55766 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47692 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51270 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47634 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:16:49 [loggers.py:248] Engine 000: Avg prompt throughput: 670.6 tokens/s, Avg generation throughput: 793.6 tokens/s, Running: 63 reqs, Waiting: 14 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 37.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55586 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55608 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55766 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55660 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45232 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:16:59 [loggers.py:248] Engine 000: Avg prompt throughput: 549.2 tokens/s, Avg generation throughput: 825.6 tokens/s, Running: 64 reqs, Waiting: 32 reqs, GPU KV cache usage: 5.4%, Prefix cache hit rate: 35.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55742 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59156 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55766 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55970 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45288 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45294 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55976 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37924 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51270 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55952 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:17:09 [loggers.py:248] Engine 000: Avg prompt throughput: 656.7 tokens/s, Avg generation throughput: 812.8 tokens/s, Running: 64 reqs, Waiting: 32 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 33.6%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59150 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55758 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45232 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47634 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39166 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37924 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:17:19 [loggers.py:248] Engine 000: Avg prompt throughput: 629.6 tokens/s, Avg generation throughput: 819.1 tokens/s, Running: 64 reqs, Waiting: 32 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 31.8%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58868 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55760 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:33280 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55660 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47618 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47634 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45294 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:17:29 [loggers.py:248] Engine 000: Avg prompt throughput: 850.0 tokens/s, Avg generation throughput: 787.2 tokens/s, Running: 63 reqs, Waiting: 27 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 30.2%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:17:39 [loggers.py:248] Engine 000: Avg prompt throughput: 950.7 tokens/s, Avg generation throughput: 787.2 tokens/s, Running: 64 reqs, Waiting: 15 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 28.1%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45288 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47692 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51270 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59148 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45294 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37924 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55958 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53630 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:17:49 [loggers.py:248] Engine 000: Avg prompt throughput: 774.6 tokens/s, Avg generation throughput: 767.6 tokens/s, Running: 64 reqs, Waiting: 28 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 26.6%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45294 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53654 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45284 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55660 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47692 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55976 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51270 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55676 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:17:59 [loggers.py:248] Engine 000: Avg prompt throughput: 676.3 tokens/s, Avg generation throughput: 812.7 tokens/s, Running: 64 reqs, Waiting: 17 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 25.5%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55688 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37924 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39112 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39122 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47674 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39166 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45258 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53654 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39158 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47692 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55976 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55660 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55782 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58910 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:18:09 [loggers.py:248] Engine 000: Avg prompt throughput: 1118.8 tokens/s, Avg generation throughput: 755.2 tokens/s, Running: 64 reqs, Waiting: 6 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 23.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55678 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47650 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45212 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47592 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58908 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53654 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56022 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55698 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55662 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55660 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56060 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:59172 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37950 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:37936 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45268 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55684 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:18:19 [loggers.py:248] Engine 000: Avg prompt throughput: 585.3 tokens/s, Avg generation throughput: 787.1 tokens/s, Running: 64 reqs, Waiting: 14 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 22.9%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56018 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39144 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45224 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55940 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47626 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53620 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45178 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56026 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55974 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53624 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45236 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45152 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51270 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55964 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45232 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53646 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57768 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45198 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57774 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56008 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56044 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57790 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57796 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56002 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57804 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57814 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:18:29 [loggers.py:248] Engine 000: Avg prompt throughput: 600.5 tokens/s, Avg generation throughput: 799.9 tokens/s, Running: 64 reqs, Waiting: 27 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 22.1%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58882 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55712 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55580 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47600 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55988 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55640 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:53614 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51256 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58892 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:47682 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45136 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:51270 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57830 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57846 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57854 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55722 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45252 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57856 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57858 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57880 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57888 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39108 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57902 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57918 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39166 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45186 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39120 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45290 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45102 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:39134 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:56032 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:58872 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:55738 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45220 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: 127.0.0.1:45164 - "POST /v1/completions HTTP/1.1" 200 OK
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:18:39 [loggers.py:248] Engine 000: Avg prompt throughput: 921.1 tokens/s, Avg generation throughput: 755.1 tokens/s, Running: 64 reqs, Waiting: 20 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 21.0%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:18:49 [loggers.py:248] Engine 000: Avg prompt throughput: 250.5 tokens/s, Avg generation throughput: 811.3 tokens/s, Running: 54 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 20.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:18:59 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 486.3 tokens/s, Running: 25 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 20.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:19:09 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 244.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 20.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:19:19 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 137.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 20.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:19:29 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 20.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:19:39 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 20.7%
|
|
[0;36m(APIServer pid=9049)[0;0m INFO 12-09 19:19:42 [launcher.py:110] Shutting down FastAPI HTTP server.
|
|
[rank0]:[W1209 19:19:42.316721556 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: Shutting down
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: Waiting for application shutdown.
|
|
[0;36m(APIServer pid=9049)[0;0m INFO: Application shutdown complete.
|