78 lines
10 KiB
Plaintext
78 lines
10 KiB
Plaintext
|
ó
|
|||
|
|
x.°iü ã ón • S SK r S SKrS SKrS SKrS SKrS SKrS SKJr \" \5 R R 5 r \" S5 r \R " S5 r\( d \" S5 \ R " S5 \R " SS5 rS 0 S
|
|||
|
|
S S .S
|
|||
|
|
.rS rS rS rS rS rS r\S:X a \" 5 gg)é N)ÚPathz/optÚLOCAL_MODEL_DIRz8Error: LOCAL_MODEL_DIR environment variable is required.é ÚVLLM_CONFIG_FILEz/etc/vllm/model_config.yamlÚ ú0.0.0.0Úinfo)ÚhostÚ log_level)ÚdefaultÚmodelsÚserverc óJ • [ R " / SQ[ R [ R SS9n U R S:X a# U R R S5 nUS:” a U$ [
|
|||
|
|
[ [ S5 R S5 5 5 $ ! N1= f! g= f) z*Detects AMD GPUs via rocm-smi or /dev/dri.)zrocm-smiz--showidz--csvT)ÚstdoutÚstderrÚtextr ÚGPUz/dev/drizrenderD*r )
|
|||
|
|
Ú
|
|||
|
|
subprocessÚrunÚPIPEÚ
|
|||
|
|
returncoder ÚcountÚlenÚlistr Úglob)Úresr s Úscripts\start_vllm.pyÚ detect_gpusr ! sŠ € ðä�nŠnÒ >ÄzÇÁÔ_i×_nÑ_nÐuyÑzˆØ �>‰>˜QÓ Ø—J‘J×$Ñ$ UÓ+ˆEØ�q‹y ˜,ðÜ”4œ˜ZÓ(×-Ñ-¨jÓ9Ó:Ó;Ð;øð ‰Dûð
|
|||
|
|
Ùús ‚A'B Á++B ÂBÂ B"c óÄ • [ R 5 n [ [ 5 R 5 ( aI [ [ SSS9 n[ R " U5 nU( a U R U5 SSS5 U $ [ S[ 35 [ S5 U $ ! , ( d f U $ = f! [ a$ n[ SU 35 [ S5 SnAU $ SnAff = f)z"Load configuration from YAML file.Úrzutf-8)ÚencodingNz%Warning: Failed to load config file: z Using default configuration.z"Warning: Config file not found at ) ÚDEFAULT_CONFIGÚcopyr Ú CONFIG_FILEÚexistsÚopenÚyamlÚ safe_loadÚupdateÚ ExceptionÚprint)ÚconfigÚfÚ user_configÚes r Ú load_configr0 1 sÄ € ä
|
|||
|
|
×
|
|||
|
|
Ñ
|
|||
|
|
Ó
|
|||
|
|
"€Fä ŒKÓ× Ñ ×!Ñ!ð 2Ü”k 3°Ò9¸QÜ "Ÿnšn¨QÓ /� Þ Ø—M‘M +Ô.÷ :ð €Mô Ð2´;°-Ð@ÔAÜ
|
|||
|
|
Ð,Ô-à €M÷ :Ô9ð €Mûô ó 2Ü Ð9¸!¸Ð=Ô >Ü Ð0× 1Ð 1ð
|
|||
|
|
€Mûð 2ús5 ¸B1 Á/B Á6B1 Â
|
|||
|
|
B.Â)B1 Â.B1 Â1
|
|||
|
|
C Â;CÃC c óH • [ R " 5 S- S- n U R 5 ( aM [ SU S3SSS9 [ R
|
|||
|
|
" SS [
|
|||
|
|
U 5 /SS
|
|||
|
|
9 U R SSS 9 [ S 5 gg! [ a n[ S
|
|||
|
|
U 35 SnAgSnAff = f)zKRemoves vLLM cache directory to fix potential graph/incompatibility issues.z.cacheÚvllmzClearing vLLM cache at ú...r T)ÚendÚflushÚrmz-rf)Úcheck)ÚparentsÚexist_okz Done.z Failed: N) r Úhomer% r+ r r ÚstrÚmkdirr* )Úcacher/ s r Únuke_vllm_cacher> D s• € ä �IŠI‹K˜(Ñ " VÑ +€EØ ‡|�|‡~�~ð #Ü Ð+¨E¨7°#Ð6¸BÀdÒ KÜ �NŠN˜D %¬¨U«Ð4¸DÒ AØ �K‰K ¨tˆKÑ 4Ü �(�Oð øô ó #Ü �I˜a˜S�/× "Ñ "ûð #ús ²A A? Á?
|
|||
|
|
B! B  B!c ó • [ R R [ U 5 n[ R R U5 ( a U$ [ R
|
|||
|
|
" [ 5 Hs nUR
|
|||
|
|
5 U R
|
|||
|
|
5 :X d M' [ R R [ U5 n[ R R U5 ( d Mq Us $ g! [ a n[ SU 35 SnAgSnAff = f)z*Get local model path from LOCAL_MODEL_DIR.zError searching for model: N) ÚosÚpathÚjoinr ÚisdirÚlistdirÚlowerr* r+ )Ú
|
|||
|
|
model_nameÚcandidate_exactÚentryÚ
|
|||
|
|
entry_pathr/ s r Úget_local_model_pathrJ P sº € ô —g‘g—l‘l¤?°JÓ?€OÜ ‡w�w‡}�}�_×%Ñ%Ø Ð ð1Ü—Z’Z¤Ö0ˆEØ�{‰{‹}
|
|||
|
|
× 0Ñ 0Ó 2Õ2Ü ŸW™WŸ\™\¬/¸5Ó A�
|
|||
|
|
Ü—7‘7—=‘= ×,Ó,Ø%Ò%ñ 1ð øô ó 1Ü
|
|||
|
|
Ð+¨A¨3Ð/×0Ð0à ûð1ús+ Á =C Â
|
|||
|
|
AC ÃC ÃC Ã
|
|||
|
|
D Ã(C;Ã;D c ój ^ • / SQn[ U 4S jU 5 5 nU( d [ ST SU 35 gg)z3Verify that the model path contains required files.)z config.jsonzpytorch_model.binzmodel.safetensorsc 3 óž ># • U HB n[ R R [ R R TU5 5 v • MD g 7f)N)r@ rA ÚisfilerB )Ú.0r- Ú
|
|||
|
|
model_paths €r Ú <genexpr>Ú$verify_model_path.<locals>.<genexpr>j s0 øé € ÐNÂXÀ”—‘—‘œrŸw™wŸ|™|¨J¸Ó :×;Ð;ÂXùs ƒA
|
|||
|
|
A
|
|||
|
|
zError: local model dir z missing expected files FT)Úanyr+ )rO ÚrequiredÚfounds ` r Úverify_model_pathrU g s8 ø€ âH€HÜ ÔNÁXÓNÓ N€EÞ Ü
|
|||
|
|
Ð'¨
|
|||
|
|
|Ð3KÈHÈ:ÐVÔWØØ ó c óœ • [ 5 n S n[ [ R 5 S:” a [ R S nU( d: U R SS5 nU( d! [ S5 [ R " S5 U R S0 5 R U5 nU( d% [ SU S35 [ R " S5 UR S5 nU( d= [ U5 nU( d+ [ S U S
|
|||
|
|
[ 35 [ R " S5 [ U5 ( d [ R " S5 [ 5 nU R S 0 5 nUR S S
|
|||
|
|
5 nSSU/nUR SU/5 SU; a UR S[ US 5 /5 UR S5 nU( a UR SU/5 SU; a+ [ US U5 n UR S[ U 5 /5 SU; a UR S[ US 5 /5 SU; a UR S[ US 5 /5 SU; a UR S [ US 5 /5 S U; a UR S US /5 OUR S S /5 S U; a UR S![ US 5 /5 S"U; a UR S#[ US" 5 /5 S$U; a UR S%[ US$ 5 /5 UR S&S'5 ( a UR S(5 UR S)S'5 ( a UR S*5 [ R R# 5 n
|
|||
|
|
UR S+S'5 ( a
|
|||
|
|
S,U
|
|||
|
|
S-' S.U
|
|||
|
|
S/' UR S0S'5 ( a
|
|||
|
|
[% 5 [ S15 [ S2U 35 [ S3U 35 [ S4U 35 SU; a [ S5US 35 U( a [ S6US S7 S8US9S 35 SU; a [ S:W 35 SU; a [ S;US 35 SU; a [ S<US 35 SU; a [ S=US 35 [ S>S?R' U5 35 [ S@5 [ R( " SXz5 g )ANr r r zJError: Either specify a model name as argument or set 'default' in config.r
|
|||
|
|
zError: Model 'z ' not found in configuration.rA zError: model 'z"' not found under LOCAL_MODEL_DIR=r r
|
|||
|
|
r r2 Úservez--hostÚportz--portÚapi_keyz --api-keyÚtensor_parallel_sizez--tensor-parallel-sizeÚ max_num_seqsz--max-num-seqsÚ
|
|||
|
|
max_model_lenz--max-model-lenÚgpu_memory_utilizationz--gpu-memory-utilizationÚdtypez--dtypeÚautoÚmax_num_batched_tokensz--max-num-batched-tokensÚ
|
|||
|
|
block_sizez --block-sizeÚ
|
|||
|
|
swap_spacez --swap-spaceÚ
|
|||
|
|
enforce_eagerFz--enforce-eagerÚ trust_remotez--trust-remote-codeÚ
|
|||
|
|
use_rocm_attnÚ1Ú$VLLM_V1_USE_PREFILL_DECODE_ATTENTIONÚ0ÚVLLM_USE_TRITON_FLASH_ATTNÚ clear_cachez=
|
|||
|
|
============================================================z Launching: z
|
|||
|
|
Model Path: z Host: z Port: z API Key: é r3 éüÿÿÿz TP Size: z Max Seqs: z Max Ctx: z GPU Util: z Command: Ú z=============================================================
|
|||
|
|
)r0 r ÚsysÚargvÚgetr+ ÚexitrJ r rU r Úextendr; ÚminÚappendr@ Úenvironr# r> rB Úexecvpe) r, rF Ú model_configrO Ú gpu_countÚ
|
|||
|
|
server_configr
|
|||
|
|
ÚcmdrZ Útp_sizeÚenvs r Úmainr~ p sm € ä
|
|||
|
|
‹]€Fð €JÜ
|
|||
|
|
Œ3�8‰8ƒ}�qÓÜ—X‘X˜a‘[ˆ
|
|||
|
|
æ Ø—Z‘Z ¨2Ó.ˆ
|
|||
|
|
ÞÜ Ð^Ô _Ü �HŠH�QŒKð —:‘:˜h¨Ó+×/Ñ/°
|
|||
|
|
Ó;€LÞ Ü
|
|||
|
|
�˜z˜lÐ*GÐHÔIÜ �Š�Œ ð ×!Ñ! &Ó)€JÞ ä)¨*Ó5ˆ
|
|||
|
|
ÞÜ �N : ,Ð.PÔQ`ÐPaÐbÔ cÜ �HŠH�QŒKô ˜Z× (Ñ (Ü �Š�Œ ô “
|
|||
|
|
€Ið —J‘J˜x¨Ó,€MØ × Ñ ˜V YÓ /€Dð �7˜JÐ
|
|||
|
|
'€Cð ‡J�J�˜$Ð Ô Ø
|
|||
|
|
� Ó Ø �
|
|||
|
|
‰
|
|||
|
|
�Hœc ,¨vÑ"6Ó 7Ð8Ô9ð × Ñ ˜yÓ)€GÞØ �
|
|||
|
|
‰
|
|||
|
|
�K Ð)Ô*ð Ó-Ü�lÐ#9Ñ:¸IÓFˆØ �
|
|||
|
|
‰
|
|||
|
|
Ð,¬c°'«lÐ;Ô<à˜ Ó%Ø �
|
|||
|
|
‰
|
|||
|
|
Ð$¤c¨,°~Ñ*FÓ&GÐHÔIà˜,Ó&Ø �
|
|||
|
|
‰
|
|||
|
|
Ð%¤s¨<¸Ñ+HÓ'IÐJÔKà <Ó/Ø �
|
|||
|
|
‰
|
|||
|
|
Ð.´°LÐAYÑ4ZÓ0[Ð\Ô]à�,Ó Ø �
|