Add Whisper language detection

[ae9is]

See #302

Adds support for automatically detecting language to Whisper tasks.

The existing HuggingFace and Whisper implementations in Python were used as reference:
Hugging Face Transformers
Original Whisper

Also updates the existing Whisper test suites, including adding a string similarity check for actual model output (as opposed to just output length). Please note that the "new" development dependency for these tests, "fastest-levenshtein" is already used by "webpack-cli".

[xenova]

Thanks for the PR! This will certainly be a useful feature. Regarding the implementation, I think it can be greatly simplified as follows:

• Instead of using .generate, perform a single forward pass of the inputs
• Then, consider all logits which correspond to the language token ids
• Choose the language with the highest score

Currently, the implementation seems to perform a full generation step (could be hundreds of forward passes).

[ae9is]

Sorry about that, it was simpler to code and the performance impact for my app was minimal! I've reworked things now to only run one pass for language detection.

Thanks for all the work on this library.

[ZhangPeng4242]

hey there, please approve this feature, it is a quite useful feature :)

[xenova]

We should be able to replace this with a single forward pass (by called this.forward(...) instead of using a generation step.

[ae9is]

There's a lot of user options for (and logic in) generate and I wanted to respect it while running language detection. It was simpler to extend generate to just stop after one pass than to duplicate that and use forward directly.

Like, hypothetically, a user adds a logits processor that suppresses the first 10 seconds worth of tokens. There is a 15s audio clip in two languages, and the context switches at 10s. The language detection should detect the second language not the first.