Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add JPQD evaluation notebook. Since JPQD QA takes about 12 hours to train, it doesn't make sense to do it in a notebook (if the browser crashes or the computer goes to sleep, training would stop). So I just refer to the example and use the notebook to evaluate the model.
This makes the notebook similar to the PTQ QA notebook. I thought about removing duplication but I think duplication in examples is not so bad, at least for now. It's nice that examples are standalone.
Since JPQD starts from a plain bert-base-uncased model I finetuned a bert-base-uncased model following the transformers run_qa.py example to compare performance.
Instead of making this a JPQD specific notebook, it could make more sense to make it a generic QA INT8 evaluation notebook, but on the other hand, it's an example, people can surely change it for similar purposes, and it's nice to promote JPQD.
TODO: the intro text at the top needs to explain a bit more about JPQD.
Colab link: https://colab.research.google.com/github/helena-intel/optimum-intel/blob/jpqd-notebook/notebooks/openvino/question_answering_quantization_jpqd.ipynb (performance is probably bad on Colab because there is no AVX512/VNNI).
@vuiseng9