Support M1/2 - With GPU / Neural Engine acceleration

[100tomer]

I don't know if it's even possible, but it would be cool and efficient to have it run on M1/2 with GPU or Neural Engine

[tomchang25]

Hi @100tomer,

Currently, whisper kernel does not support the GPU acceleration library for Apple, which is far beyond my capabilities.
However, I have learned that there is a project underway to address this issue.

As an alternative solution, I will release a light-weight version(#19) that will achieve satisfactory results by compressed audio and through OpenAI API integration.
This should be completed in the near future and included in version 3.1.

[100tomer]

Hi @100tomer,

Currently, whisper kernel does not support the GPU acceleration library for Apple, which is far beyond my capabilities.

However, I have learned that there is a project underway to address this issue.

As an alternative solution, I will release a light-weight version(#19) that will achieve satisfactory results by compressed audio and through OpenAI API integration.

This should be completed in the near future and included in version 3.1.

Thanks, sounds great

[Hans-han]

it is actually not faster with GPU. there is a cpp version of Whisper which is also on GitHub. The author tested the GPU vs CPU and found it is close in terms of performence . his guess was the bandwidth is the limitation.

[100tomer]

it is actually not faster with GPU. there is a cpp version of Whisper which is also on GitHub. The author tested the GPU vs CPU and found it is close in terms of performence . his guess was the bandwidth is the limitation.

Oh that's the ram bandwidth? Or the SSD bandwidth? Also does it use GPU only or the Neural Engine too?