Create whisper_evaluator.py #3990
Conversation
|
@eaidova, please take a look and trigger the CI please |
| return [], outputs | ||
|
|
||
|
|
||
| class GenAI_WhisperPipeline(WhisperPipeline): |
There was a problem hiding this comment.
I would rename classes for consistency, e.g. HFWhisperPipeline, OptimumWhisperPipeline, GenAIWhisperPipeline,
There was a problem hiding this comment.
Agree - suggested names are self descriptive
|
it looks good overall. It would be great to get some sanity tests on a dummy model to make sure that all three classes work. |
| import openvino_genai as ov_genai | ||
| from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor | ||
| from transformers.pipelines.automatic_speech_recognition import \ | ||
| AutomaticSpeechRecognitionPipeline |
There was a problem hiding this comment.
please make these packages optionl like inflect bellow
There was a problem hiding this comment.
Agree. I had them all in try except in initial version but then I thought that packages are so common that was no sense to import, but indeed there are checks that fails to import them though.
e.g. with |
| input_data = [sample["audio"]["array"]] | ||
| input_meta = [{"sample_rate": sample["audio"]["sampling_rate"]}] | ||
| identifiers = [sample["id"]] | ||
| # print(ground_truth) |
There was a problem hiding this comment.
Please clean up the code a bit and remove print. Also, you need to remove the directory after test suite finishes. You can define teardown_module() function for that.
There was a problem hiding this comment.
also looks like you need to install datasets in test requirements
tools/accuracy_checker/tests/test_whisper_evaluator.py:23: in <module>
from datasets import load_dataset
E ModuleNotFoundError: No module named 'datasets'
There was a problem hiding this comment.
@pwolnows I believe you have enough permissions to open github actions status, right? still some dependencies missed
https://github.com/openvinotoolkit/open_model_zoo/actions/runs/12442951517/job/34781160595?pr=3990
Enable validation of whisper models with: