-
|
I've seen it mentioned in the last release but there is no mention of it on the wiki nor I could find anything relevant within issues. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 4 replies
-
|
It's a more efficient algorithm that is generally faster and less memory intensive |
Beta Was this translation helpful? Give feedback.
-
|
So... should we enable it on every gguf model? Any downsides? |
Beta Was this translation helpful? Give feedback.
-
|
Poor performance on older cards |
Beta Was this translation helpful? Give feedback.
-
|
What counts as an "older card" 😆? I added this flag for a quick test, and on my AMD Radeon RX 7900 XTX the token generation speed tanked by at least 75%. |
Beta Was this translation helpful? Give feedback.
-
|
I guess yours counts as an "older card" then lol. Jokes aside I noticed that amd cards are usually hit or miss when it comes to the latest optimizations. |
Beta Was this translation helpful? Give feedback.
I guess yours counts as an "older card" then lol. Jokes aside I noticed that amd cards are usually hit or miss when it comes to the latest optimizations.