Brief introduction and discussion of Pre-requisite task. [GSOC]

[dwipddalal]

Hi OpenVINO Developers,

I am Dwip Dalal, B.Tech Junior Undergraduate at the prestigious Indian Institute of Technology (IIT), Gandhinagar. Over the last 3.5 years, I have been working extensively in the field of machine learning, with a particular focus on deep learning. I have had the privilege of being associated with the OpenVINO community for the past year and have contributed to the community in the past as well 223-gpt2-text-prediction. Apart from this, I have used OpenVINO toolkit many times in different projects for performing quantization and pruning. Some of the projects in which I used the OpenVINO toolkit are noted in the following paragraph.

In my work Deep learning career of the last 3.5 years, I have done lot of industrial internships and research projects (working with the Univerisity of British Columbia, the University of South Carolina, TCS Research, ISRO, DRDO, and many other startups). One of the interns was with a computer vision startup named AIVID TECHVISION, where I made a video anomaly detection model for CCTV camera feed. In this project, I used OpenVINO toolkit for pruning it before deployment for real-time detection. Another was a research project for building a smart surveillance drone. In this project, also I used the OpenVINO toolkit for quantization and pruning of models of video and image processing before deployment on drones, which was later awarded as the best research work of the year by the Director of IIT Gandhinagar.

I am highly excited to spend this summer contributing to OpenVINO. I am excited for two projects:
2. Showcase performance of PyTorch Image Models (timm) with OpenVINO
4. Smart meeting notes

Before getting started, I wanted to inquire if there are any additional prerequisites for these projects besides the one already posted on the OpenVINO wiki page. I am highly enthusiastic about these projects and would be grateful to discuss them further. Thank you for your valuable time and consideration, and I look forward to hearing from you soon.

[AlexKoff88]

Hi, Dwip Dalal! It is a pleasure to hear about your experience with OpenVINO. Can you please briefly describe how you used quantization and pruning with OpenVINO. What kind of API did you use to quantize or/and prune the model? What were the performance improvements that you got with this method for your models?

[AlexKoff88]

Any public links to the scripts where the pruning and quantization flows are implemented would be beneficial.

[dwipddalal]

Hi @AlexKoff88,
It's nice to chat with you.
In both the projects that I mentioned above, I had to maintain a specific threshold value of the metrics that we were using for those cases. In order to accomplish that, I made a pipeline, i.e., first, the trained model will be quantized using POT (Post training quantization) API. If the metric value of the resultant model is still above the threshold, then take the next step to prune the model further using the Filter pruning technique using NNCF. If the accuracy obtained after applying POT 8-bit integer quantization is below the threshold, then use NNCF quantization API.

Performances improvements are:

For Shoplifting detection Project (Internship that I did at AIVID TECHVISION)

• I was able to achieve 60% drop in inference time and 52% drop in storage space with 2.7% drop in F1 score (the metric used in that project)

For smart surveillance Drone project - stats are as follows:

All these three models were made by me and then deployed on Drone using the Jetson Nano board.

Unfortunately, since these scripts were part of my internship, so I don't currently have access to them. But I do have some scripts that I made for some other tasks. For instance, I have attached one script with this message which was developed by me in March 2022 for getting the AIVID TECHVISION internship (as they had kept an internship test in which we had to use OpenVINO and quantize Inception V3 model).

Thanks for your time and consideration. Hoping to hear from you soon.

quantization inception.zip

[AlexKoff88]

Hi,

That is a really great experience! Thanks for sharing it.
cc'ed @l-bat

As a pretest task you can:

• Take one of the vision transformer model from TIMM, let’s say “swin_small_patch4_window7_224”.
• Try PyTorch built-in quantization to quantize the model
• A simple Python script would be nice to have as well as reported performance gain after quantization with either TorchScript, ONNXRuntime or OpenVINO and model size reduction.
• As an advance task. You can play with the bitwidth and quantize weights to 4 bit.

Here are some materials to study that can help you:

• https://pytorch.org/docs/stable/quantization.html
• https://github.com/pytorch/pytorch/blob/05943712a443138497c185405b575043b2916f34/torch/ao/quantization/qconfig.py#L58
• https://github.com/pytorch/pytorch/blob/1e2e14957090d7bc16b8373537422a690b05a6c0/torch/ao/quantization/fake_quantize.py#L410

[dwipddalal]

Hi,
Sure! Seems interesting. I shall start working on it as soon as possible.
Thanks for sharing the reference materials.

[dwipddalal]

Hi @AlexKoff88
I hope this message finds you well. I am writing to update you on my progress with the GSOC program. I am pleased to inform you that I have completed both the general pre-requisite task (made the PR for OpenVINO notebook repository) and the pretest task that you assigned me. I have also written the proposal for the problem statement and uploaded it.

I want to apologize for not being able to reach out to you earlier. Unfortunately, I had my final exams and then was hospitalized due to an accident, which delayed my response. However, I am now fully recovered and ready to dive into the GSOC program.

I am eager to hear your feedback on my proposal and hope that I can have the opportunity to work under your guidance. Thank you for your time and consideration. I look forward to hearing from you soon.

[AlexKoff88]

Thank you! I will review the proposal soon.