Release v2.0.1 of NNCF to master

.gitattributes

@@ -1,4 +1,5 @@
 *.png filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
 *.pb filter=lfs diff=lfs merge=lfs -text
 
 * text=auto eol=lf

.github/labeler.yml

@@ -0,0 +1,5 @@
+# See help here: https://github.com/marketplace/actions/labeler
+
+dependencies:
+  - '*requirements*'
+  - '*setup.py'

.github/pull_request_template.md

@@ -0,0 +1,15 @@
+### Changes
+
+<!--- What was changed (briefly), how to reproduce (if applicable), what the reviewers should focus on -->
+
+### Reason for changes
+
+<!--- Why should the change be applied -->
+
+### Related tickets
+
+<!--- Post the numerical ID of the ticket, if available -->
+
+### Tests
+
+<!--- How was the correctness of changes tested and whether new tests were added -->

.github/workflows/labeler.yml

@@ -0,0 +1,11 @@
+name: "Pull Request Labeler"
+on: [pull_request_target]
+
+jobs:
+  triage:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/labeler@v3
+        with:
+          repo-token: "${{ secrets.GITHUB_TOKEN }}"
+          sync-labels: true

README.md

@@ -141,10 +141,10 @@ See [third_party_integration](./third_party_integration) for examples of code mo
 - Ubuntu\* 18.04 or later (64-bit)
 - Python\* 3.6.2 or later
 - Supported frameworks:
-  - PyTorch\* >=1.5.0, <=1.8.1 (1.8.0 not supported)
-  - TensorFlow\* 2.4.2
+  - PyTorch\* >=1.5.0, <=1.9.1 (1.8.0 not supported)
+  - TensorFlow\* 2.4.3
 
-This repository is tested on Python* 3.6.2+, PyTorch* 1.8.1 (NVidia CUDA\* Toolkit 10.2) and TensorFlow* 2.4.2 (NVidia CUDA\* Toolkit 11.0).
+This repository is tested on Python* 3.6.2+, PyTorch* 1.9.1 (NVidia CUDA\* Toolkit 10.2) and TensorFlow* 2.4.3 (NVidia CUDA\* Toolkit 11.0).
 
 ## Installation
 We suggest to install or use the package in the [Python virtual environment](https://docs.python.org/3/tutorial/venv.html).

ReleaseNotes.md

@@ -7,6 +7,29 @@ samples distributed with the code.  The samples demonstrate the usage of compres
 public models and datasets for three different use cases: Image Classification, Object Detection,
 and Semantic Segmentation.
 
+## New in Release 2.0.1
+Target version updates:
+- Bump target framework versions to PyTorch 1.9.1 and TensorFlow 2.4.3
+- Increased target HuggingFace transformers version for the integration patch to 4.9.1
+
+Bugfixes:
+- Fixed statistic collection for the algo mixing scenario
+- Increased pruning algorithm robustness in cases of a disconnected NNCF graph
+- Fixed the fatality of NNCF graph PNG rendering failures
+- Fixed README command lines
+- (PyTorch) Fixed a bug with quantizing shared weights multiple times
+- (PyTorch) Fixed knowledge distillation failures in CPU-only and DataParallel scenarios
+- (PyTorch) Fixed sparsity application for torch.nn.Embedding and EmbeddingBag modules
+- (PyTorch) Added GroupNorm + ReLU as a fusable pattern
+- (TensorFlow) Fixed gamma fusion handling for pruning TF BatchNorm
+- (PyTorch) Fixed pruning for models where operations have multiple convolution predecessors
+- (PyTorch) Fixed NNCFNetwork wrapper so that `self` in the calls to the wrapped model refer to the wrapper NNCFNetwork object and not to the wrapped model
+- (PyTorch) Fixed tracing of `view` operations to handle shape arguments with the `torch.Tensor` type 
+- (PyTorch) Added matmul ops to be considered for fusing
+- (PyTorch, TensorFlow) Fixed tensorboard logging for accuracy-aware scenarios
+- (PyTorch, TensorFlow) Fixed FLOPS calculation for grouped convolutions
+- (PyTorch) Fixed knowledge distillation failures for tensors of unsupported shapes - will ignore output tensors with unsupported shapes now instead of crashing.
+
 ## New in Release 2.0:
 - Added TensorFlow 2.4.2 support - NNCF can now be used to apply the compression algorithms to models originally trained in TensorFlow.
 NNCF with TensorFlow backend supports the following features:

docs/Usage.md

@@ -342,4 +342,4 @@ model = training_loop.run(model,
                           validate_fn=validate_fn,
                           configure_optimizers_fn=configure_optimizers_fn)
 ```
-The above call executes the acccuracy-aware adaptive compression training loop and return the compressed model with the maximal found compression rate and satisfying the defined accuracy drop criteria. For more details on how to use the accuracy-aware training loop functionality of NNCF, please refer to its [documentation](./accuracy_aware_model_training/TrainingLoop.md).
+The above call executes the acccuracy-aware adaptive compression training loop and return the compressed model with the maximal found compression rate and satisfying the defined accuracy drop criteria. For more details on how to use the accuracy-aware training loop functionality of NNCF, please refer to its [documentation](./accuracy_aware_model_training/AdaptiveCompressionTraining.md).

docs/accuracy_aware_model_training/AdaptiveCompressionTraining.md

@@ -1,6 +1,6 @@
 # Accuracy-aware training loop in NNCF
 
-To launch the adaptive compression training loop, the user is expected to define several function related to model training, validation and optimizer creation (see [the usage documentation](./docs/Usage.md#accuracy-aware-model-training) for more details) and pass them to the run method of an `AdaptiveCompressionTrainingLoop` instance. The training loop logic inside of the `AdaptiveCompressionTrainingLoop` is framework-agnostic, while all of the framework specifics are encapsulated inside of corresponding `Runner` objects, which are created and called inside the training loop. The adaptive compression training loop is generally aimed at automatically searching for the optimal compression rate in the model, with the parameters of the search algorithm specified in the configuration file as follows:
+To launch the adaptive compression training loop, the user is expected to define several function related to model training, validation and optimizer creation (see [the usage documentation](../Usage.md#accuracy-aware-model-training) for more details) and pass them to the run method of an `AdaptiveCompressionTrainingLoop` instance. The training loop logic inside of the `AdaptiveCompressionTrainingLoop` is framework-agnostic, while all of the framework specifics are encapsulated inside of corresponding `Runner` objects, which are created and called inside the training loop. The adaptive compression training loop is generally aimed at automatically searching for the optimal compression rate in the model, with the parameters of the search algorithm specified in the configuration file as follows:
 ```
 "compression": [
     {

docs/compression_algorithms/Sparsity.md

@@ -52,7 +52,7 @@ sparsity and filter pruning algorithms. It can be enabled by setting a non-zero
             "sparsity_target_epoch": 3, // Index of the epoch from which the sparsity level of the model will be equal to spatsity_target value
             "sparsity_freeze_epoch": 50, // Index of the epoch from which the sparsity mask will be frozen and no longer trained
             "multistep_steps": [10, 20], // A list of scheduler steps at which to transition to the next scheduled sparsity level (multistep scheduler only).
-            "multistep_sparsity_levels": [0.2, 0.5] //Levels of sparsity to use at each step of the scheduler as specified in the 'multistep_steps' attribute. The firstsparsity level will be applied immediately, so the length of this list should be larger than the length of the 'steps' by one."
+            "multistep_sparsity_levels": [0.2, 0.5, 0.7] // Levels of sparsity to use at each step of the scheduler as specified in the 'multistep_steps' attribute. The first sparsity level will be applied immediately, so the length of this list should be larger than the length of the 'steps' by one. The last sparsity level will function as the ultimate sparsity target, overriding the "sparsity_target" setting if it is present.
     },
 
     // A list of model control flow graph node scopes to be ignored for this operation - functions as a 'denylist'. Optional.
@@ -88,7 +88,7 @@ The magnitude sparsity method implements a naive approach that is based on the a
             "sparsity_target_epoch": 3, // Index of the epoch from which the sparsity level of the model will be equal to spatsity_target value
             "sparsity_freeze_epoch": 50, // Index of the epoch from which the sparsity mask will be frozen and no longer trained
             "multistep_steps": [10, 20], // A list of scheduler steps at which to transition to the next scheduled sparsity level (multistep scheduler only).
-            "multistep_sparsity_levels": [0.2, 0.5] //Levels of sparsity to use at each step of the scheduler as specified in the 'multistep_steps' attribute. The firstsparsity level will be applied immediately, so the length of this list should be larger than the length of the 'steps' by one."
+            "multistep_sparsity_levels": [0.2, 0.5, 0.7] // Levels of sparsity to use at each step of the scheduler as specified in the 'multistep_steps' attribute. The first sparsity level will be applied immediately, so the length of this list should be larger than the length of the 'steps' by one. The last sparsity level will function as the ultimate sparsity target, overriding the "sparsity_target" setting if it is present.
     },
 
     // A list of model control flow graph node scopes to be ignored for this operation - functions as a 'denylist'. Optional.

examples/tensorflow/classification/README.md

@@ -119,7 +119,7 @@ To export a model to the OpenVINO IR and run it using the Intel® Deep Learning
 |MobileNet V3 small|INT8 (per-channel, symmetric for weights; per-tensor, asymmetric for activations) |ImageNet|67.7 (0.68)|[mobilenet_v3_small_imagenet_int8.json](configs/quantization/mobilenet_v3_small_imagenet_int8.json)|[Link](https://storage.openvinotoolkit.org/repositories/nncf/models/v2.0.0/tensorflow/mobilenet_v3_small_int8_w_sym_ch_half_a_asym_t.tar.gz)|
 |MobileNet V3 small|INT8 (per-channel, symmetric for weights; per-tensor, asymmetric for activations) + Sparsity 42% (RB)|ImageNet|67.7 (0.68)|[mobilenet_v3_small_imagenet_rb_sparsity_int8.json](configs/sparsity_quantization/mobilenet_v3_small_imagenet_rb_sparsity_int8.json)|[Link](https://storage.openvinotoolkit.org/repositories/nncf/models/v2.0.0/tensorflow/mobilenet_v3_small_int8_w_sym_ch_half_a_asym_t_rb_sparsity_42.tar.gz)|
 |MobileNet V3 large|INT8 (per-channel, symmetric for weights; per-tensor, asymmetric for activations) |ImageNet|75.0 (0.81)|[mobilenet_v3_large_imagenet_int8.json](configs/quantization/mobilenet_v3_large_imagenet_int8.json)|[Link](https://storage.openvinotoolkit.org/repositories/nncf/models/v2.0.0/tensorflow/mobilenet_v3_large_int8_w_sym_ch_half_a_asym_t.tar.gz)|
-|MobileNet V3 large|INT8 (per-channel, symmetric for weights; per-tensor, asymmetric for activations) + Sparsity 42% (RB)|ImageNet|75.15 (0.66)|[mobilenet_v3_large_imagenet_rb_sparsity_int8.json](configs/sparsity_quantization/mobilenet_v3_large_imagenet_rb_sparsity_int8.json)|[Link](https://storage.openvinotoolkit.org/repositories/nncf/models/v2.0.0/tensorflow/mobilenet_v3_large_int8_w_sym_ch_half_a_asym_t_sparsity_42.tar.gz)|
+|MobileNet V3 large|INT8 (per-channel, symmetric for weights; per-tensor, asymmetric for activations) + Sparsity 42% (RB)|ImageNet|75.15 (0.66)|[mobilenet_v3_large_imagenet_rb_sparsity_int8.json](configs/sparsity_quantization/mobilenet_v3_large_imagenet_rb_sparsity_int8.json)|[Link](https://storage.openvinotoolkit.org/repositories/nncf/models/v2.0.0/tensorflow/mobilenet_v3_large_int8_w_sym_ch_half_a_asym_t_rb_sparsity_42.tar.gz)|
 |ResNet50|INT8 (per-tensor, symmetric for weights; per-tensor, symmetric for activations)|ImageNet|75.0 (0.04)|[resnet50_imagenet_int8.json](configs/quantization/resnet50_imagenet_int8.json)|[Link](https://storage.openvinotoolkit.org/repositories/nncf/models/v2.0.0/tensorflow/resnet50_int8_w_sym_t_half_a_sym_t.tar.gz)|
 |ResNet50|Sparsity 80% (RB)|ImageNet|74.36 (0.68)|[resnet50_imagenet_rb_sparsity.json](configs/sparsity/resnet50_imagenet_rb_sparsity.json)|[Link](https://storage.openvinotoolkit.org/repositories/nncf/models/v2.0.0/tensorflow/resnet50_rb_sparsity_80.tar.gz)|
 |ResNet50|INT8 (per-tensor, symmetric for weights; per-tensor, symmetric for activations) + Sparsity 65% (RB)|ImageNet|74.3 (0.74)|[resnet50_imagenet_rb_sparsity_int8.json](configs/sparsity_quantization/resnet50_imagenet_rb_sparsity_int8.json)|[Link](https://storage.openvinotoolkit.org/repositories/nncf/models/v2.0.0/tensorflow/resnet50_int8_w_sym_t_half_a_sym_t_rb_sparsity_65.tar.gz)|

examples/tensorflow/classification/main.py

@@ -43,6 +43,7 @@
 from examples.tensorflow.common.utils import serialize_config
 from examples.tensorflow.common.utils import serialize_cli_args
 from examples.tensorflow.common.utils import write_metrics
+from examples.tensorflow.common.utils import SummaryWriter
 
 
 def get_argument_parser():
@@ -256,7 +257,8 @@ def run(config):
                                               callbacks=callbacks,
                                               initial_epoch=initial_epoch,
                                               steps_per_epoch=train_steps,
-                                              tensorboard_writer=config.tb,
+                                              tensorboard_writer=SummaryWriter(config.log_dir,
+                                                                               'accuracy_aware_training'),
                                               log_dir=config.log_dir,
                                               uncompressed_model_accuracy=uncompressed_model_accuracy,
                                               result_dict_to_val_metric_fn=result_dict_to_val_metric_fn,

examples/tensorflow/common/callbacks.py

@@ -83,8 +83,9 @@ def __init__(self,
         self.step = initial_step
         self._track_lr = track_lr
 
+    # pylint: disable=W0237
     def on_train_batch_begin(self,
-                             epoch: int, # pylint: disable=W0613
+                             epoch: int,
                              logs: MutableMapping[str, Any] = None) -> None:
         self.step += 1
         logs = logs or {}

examples/tensorflow/common/object_detection/losses.py

@@ -698,7 +698,7 @@ def yolo3_decode(self, feats, anchors, num_classes, input_shape, scale_x_y=None)
 
     def get_anchors(self, anchors_path):
         """loads the anchors from a file"""
-        with open(anchors_path) as f:
+        with open(anchors_path, encoding='utf8') as f:
             anchors = f.readline()
         anchors = [float(x) for x in anchors.split(',')]
         return np.array(anchors).reshape(-1, 2)

examples/tensorflow/common/object_detection/utils/box_coder.py

@@ -13,7 +13,6 @@
 
 from abc import ABCMeta
 from abc import abstractmethod
-from abc import abstractproperty
 
 import tensorflow as tf
 
@@ -28,7 +27,8 @@ class BoxCoder:
     """Abstract base class for box coder."""
     __metaclass__ = ABCMeta
 
-    @abstractproperty
+    @property
+    @abstractmethod
     def code_size(self):
         """Return the size of each code.
 

examples/tensorflow/common/sample_config.py

@@ -74,7 +74,7 @@ def parse_known_args(self, args=None, namespace=None):
 class SampleConfig(Dict):
     @classmethod
     def from_json(cls, path) -> 'SampleConfig':
-        with open(path) as f:
+        with open(path, encoding='utf8') as f:
             loaded_json = json.load(f)
         return cls(loaded_json)
 

examples/tensorflow/common/utils.py

@@ -65,7 +65,7 @@ def get_name(config):
 def write_metrics(acc, filename):
     avg = round(acc * 100, 2)
     metrics = {"Accuracy": avg}
-    with open(filename, 'w') as outfile:
+    with open(filename, 'w', encoding='utf8') as outfile:
         json.dump(metrics, outfile)
 
 
@@ -116,7 +116,7 @@ def print_args(config, logger=default_logger):
 
 
 def serialize_config(config, log_dir):
-    with open(osp.join(log_dir, 'config.json'), 'w') as f:
+    with open(osp.join(log_dir, 'config.json'), 'w', encoding='utf8') as f:
         json.dump(config, f, indent=4)
 
 
@@ -126,7 +126,7 @@ def serialize_cli_args(argparser, argv, log_dir):
         cli_args = {k:v for k, v in vars(args).items() if k in argparser.seen_actions}
     else:
         cli_args = {k:v for k, v in vars(args).items() if v is not None}
-    with open(osp.join(log_dir, 'cli_args.json'), 'w') as f:
+    with open(osp.join(log_dir, 'cli_args.json'), 'w', encoding='utf8') as f:
         json.dump(cli_args, f, indent=4)
 
 

examples/tensorflow/object_detection/main.py

@@ -358,7 +358,8 @@ def validate_fn(model, **kwargs):
             compress_model = acc_aware_training_loop.run(compress_model,
                                                          train_epoch_fn=train_epoch_fn,
                                                          validate_fn=validate_fn,
-                                                         tensorboard_writer=config.tb,
+                                                         tensorboard_writer=SummaryWriter(config.log_dir,
+                                                                                          'accuracy_aware_training'),
                                                          log_dir=config.log_dir)
         else:
             train(train_step, test_step, eval_metric, train_dist_dataset, test_dist_dataset,

examples/tensorflow/object_detection/models/yolo_v4_model.py

@@ -72,7 +72,7 @@ def build_model(self, weights=None, is_training=None):
             keras_model.load_weights(weights, by_name=True)
         return keras_model
 
-    def build_loss_fn(self, compress_model, compression_loss_fn):
+    def build_loss_fn(self, keras_model, compression_loss_fn):
         def _total_loss_fn(labels, outputs):
             loss_fn_out = self._loss_fn(labels, outputs,
                                         self._params.anchors,

examples/tensorflow/object_detection/postprocessing/yolo_v4_postprocessing.py

@@ -510,7 +510,7 @@ def yolo_adjust_boxes(boxes, img_shape):
 
 def get_anchors(anchors_path):
     """loads the anchors from a file"""
-    with open(anchors_path) as f:
+    with open(anchors_path, encoding='utf8') as f:
         anchors = f.readline()
     anchors = [float(x) for x in anchors.split(',')]
     return np.array(anchors).reshape(-1, 2)