vllm or something
#25
by prudant - opened
nice work, can be served for production environments with vllm or similars? regards
Tried with vllm/vllm-openai:cu129-nightly
Currently does not work for 5090. RTX Pro 6000, GB10
Require Flash Attention 3 to work (H100, H200...)
Tried with
vllm/vllm-openai:cu129-nightly
Currently does not work for 5090. RTX Pro 6000, GB10
Require Flash Attention 3 to work (H100, H200...)
Try with TRITON BACKEND instead of FA3
You would know it won't work if you tried, for this model, vllm's implementation requires you to use FA3.