New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

I encountered the same issue while using `baichuan2-13B-chat`.. #2780

Open

Lacacy opened this issue Nov 26, 2024 · 0 comments · May be fixed by #2798

Lacacy commented Nov 26, 2024

          I encountered the same issue while using `baichuan2-13B-chat`..

I extracted the chat parameters from baichuan2's generation_config.json, and when I call the tgi interface, the result is as follows.

When I invoke the chat method, the result is as follows.

Here are the deployment parameters for the tgi.

--max-batch-prefill-tokens 4096 --max-input-length 4096 --max-total-tokens 4608

Originally posted by @zTaoplus in #981 (comment)

The text was updated successfully, but these errors were encountered:

Lacacy linked a pull request

that will close this issue

Update tensor_parallel.py #2798

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment