random text generation from Qwen2-VL-7B-Instruct with TGI3

[DongyoungKim2]

System Info

OS version: Ubuntu 22.04
Model: Qwen2-VL-7B-Instruct
GPU: H100 80G
Deployment: k8s

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

I compared output between vLLM and TGI3 and TGI3 produce random text for non-english cases

1. run Qwen2-VL-7B-Instruct model using tgi:3.0.0image
2. run the same model using vllm:v0.6.4.post1 image
3. for TGI3, no cuda graph used (Error for Qwen2-VL-2B-Instruct using v3.0.0 #2823)

and do the inference with openai api and TGI3 produce non-sense random characters.
Task was OCR for given image (written in Korean) and vLLM got correctly but TGI3.

Expected behavior

output from vLLM and TGI3 should be similar