New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Load model weight fast #2836

Open

Zzzz1111 opened this issue Dec 13, 2024 · 2 comments

Zzzz1111 commented Dec 13, 2024

Feature request

Accelerate model weight loading，Now it takes a lot of time

Motivation

When I start tgi, it takes up most of my time

Your contribution

I have modified my local tgi and want to submit PR

Author

Zzzz1111 commented Dec 13, 2024

Hi,my idea is different,what I do is load the weights directly to the corresponding device; when i using cuda, this way will make it fast

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment