OTX D-Fine Detection Algorithm Integration #4142

eugene123tw · 2024-12-04T11:09:14Z

Summary

OTX D-Fine Detection Algorithm Integration: https://github.com/Peterande/D-FINE

Introduced five variants of the D-Fine detection algorithm.
Integrated the HGNetv2 backbone from PaddleDetection.
Cleaned and optimized the original codebase by:
- Reducing code duplication where possible.
- Adding docstrings for all methods and functions.
- Benchmarking OpenVINO/PyTorch detection results for accuracy and performance.

Next phase

Validate potential module combinations that could be unified in future iterations, such as:
- D-Fine Decoder and RT-DETR Decoder.
- D-Fine Hybrid Encoder and RT-DETR Decoder.
- D-Fine Criterion and RT-DETR Criterion.
Validate Post-Training Optimization Tool (POT) results and assess potential accuracy drops.
Validate XAI feature.

How to test

otx train --config src/otx/recipe/detection/dfine_x.yaml --data_root DATA_ROOT
pytest tests/unit/algo/detection/test_dfine.py

Checklist

I have added unit tests to cover my changes.
I have added integration tests to cover my changes.
I have ran e2e tests and there is no issues.
I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).
I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
I have linked related issues.

License

I submit my code changes under the same Apache License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below).

# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

…onfiguration

…FINETransformerModule * Refactor DFineCriterion

… redundant parameters

…nd reorganizing imports

…s, and updating documentation

…es, and enhancing documentation for RandomIoUCrop

…tructure and updating type hints in DFINECriterion

…ng parameter names for consistency

…ion documentation

kprokofi

Thank you, Eugene for your great contribution!
I will try D-Fine from your branch with Intel GPUs

kprokofi · 2024-12-23T10:42:52Z

src/otx/algo/detection/heads/dfine_decoder.py

+    return output.permute(0, 2, 1)
+
+
+class MSDeformableAttentionV2(nn.Module):


Can we use this for RTDetr as well? Maybe it will be upgrade for RTDetrV2

Secondly, I would rather put it to otx/src/otx/algo/common/layers/transformer_layers.py as done for RTDetr.

kprokofi · 2024-12-23T10:47:25Z

src/otx/algo/detection/d_fine.py

+
+PRETRAINED_ROOT: str = "https://github.com/Peterande/storage/releases/download/dfinev1.0/"
+
+PRETRAINED_WEIGHTS: dict[str, str] = {


I wonder whether we need all of these variants? We are currently overwhelmed with detection recipes. Could we choose maybe 2 models to expose and omit others? The largest one shows the best performance and it is a candidate for Geti largest template revamp, but other templates seems to be not so beneficial comparing with already introduced models.
So, I would consider cleaning some model versions here (same concerns RTDetr and YOLOX, but it is another story)

kprokofi · 2024-12-23T10:51:30Z

src/otx/algo/detection/heads/dfine_decoder.py

+    )
+
+
+def distance2bbox(points: Tensor, distance: Tensor, reg_scale: Tensor) -> Tensor:


maybe put this to utils?

kprokofi · 2024-12-23T10:52:56Z

src/otx/algo/detection/heads/dfine_decoder.py

+    return box_convert(bboxes, in_fmt="xyxy", out_fmt="cxcywh")
+
+
+def deformable_attention_core_func_v2(


Same comment about the location, maybe: otx/src/otx/algo/modules/transformer.py?

kprokofi · 2024-12-23T10:53:56Z

src/otx/algo/detection/necks/dfine_hybrid_encoder.py

+class HybridEncoderModule(nn.Module):
+    """HybridEncoder for DFine.
+
+    TODO(Eugene): Merge with current rtdetr.HybridEncoderModule in next PR.


kprokofi · 2024-12-23T10:56:26Z

src/otx/core/data/transform_libs/torchvision.py

@@ -3921,3 +3921,44 @@ def _dispatch_transform(cls, cfg_transform: DictConfig | dict | tvt_v2.Transform
            raise TypeError(msg)

        return transform
+
+
+class RandomIoUCrop(tvt_v2.RandomIoUCrop):


If we use already defined RandomIOUCrop in this file, the performance issues occur?

eugene123tw added 30 commits November 26, 2024 14:48

init

159c2fc

remove convertbox

68b6a0d

Refactor D-FINE detector: remove unused components and update model c…

7a35df4

…onfiguration

update

c0d4f99

update

8da83fa

Update

2bd71ca

update recipes

a1da0e1

Add d-fine-m

1f2a9b3

Fix recipes

f5aa07c

Merge branch 'develop' into eugene/d-fine-poc

8bf6df2

dfine-l

84baf23

Add dfine m - no aug

adf6947

format changes

3d92e7e

learnable params + disable teacher distillation

b91bb42

update

7daecda

add recipes

8724219

update

5e0f0f7

update

2140f68

update recipes

71ce5d3

add dfine_hgnetv2_x

175fc07

Update recipes

5b7f4aa

add tile DFine recipes

4d523f6

update recipes and tile batch size

c40190a

update

6b12cb3

Merge branch 'develop' into eugene/d-fine-poc

090b2ca

update LR

4381600

DFine revert LR changes

a305240

make multi-scale optional

2149406

update tile recipes

62d63a0

update tiling recipes

11679a9

eugene123tw added 2 commits December 17, 2024 18:33

* Fix docstring punctuation and remove unused aux_loss parameter in D…

f9d7137

…FINETransformerModule * Refactor DFineCriterion

Update style changes

021cf2f

github-actions bot added the TEST Any changes in tests label Dec 18, 2024

eugene123tw added 10 commits December 18, 2024 21:13

conv batchnorm fuse

7ee9fa2

update hybrid encoder

a88a9d1

Refactor DFINE HybridEncoderModule to improve code clarity and remove…

edf22a0

… redundant parameters

minor update

53c635d

Refactor D-FINE module structure by removing obsolete detector file a…

fe1e7f9

…nd reorganizing imports

Refactor import paths in D-FINE module and clean up unused code

45d0360

Refactor D-FINE module by removing commented code, cleaning up import…

7b4c899

…s, and updating documentation

Refactor D-FINE module by updating type hints, improving error messag…

0d0f99b

…es, and enhancing documentation for RandomIoUCrop

Refactor D-FINE module by improving the weighting function's return s…

64ea42a

…tructure and updating type hints in DFINECriterion

Update d-fine unit test

dff6ecc

eugene123tw changed the title ~~[Draft] D-Fine PoC~~ D-Fine Detection Algorithm Dec 20, 2024

eugene123tw marked this pull request as ready for review December 20, 2024 15:32

eugene123tw requested review from samet-akcay, kprokofi, sovrasov, negvet, Daankrol, djdameln, ashwinvaidya17, rajeshgangireddy and atwinand as code owners December 20, 2024 15:32

eugene123tw changed the title ~~D-Fine Detection Algorithm~~ OTX D-Fine Detection Algorithm Integration Dec 20, 2024

eugene123tw added 2 commits December 20, 2024 15:56

Refactor D-FINE module by enhancing docstrings for clarity and updati…

b676b51

…ng parameter names for consistency

Add D-Fine Detection Algorithm entries to CHANGELOG and object detect…

76089f1

…ion documentation

github-actions bot added the DOC Improvements or additions to documentation label Dec 20, 2024

kprokofi reviewed Dec 23, 2024

View reviewed changes

Fix device assignment for positional embeddings in HybridEncoderModule

a6c330e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OTX D-Fine Detection Algorithm Integration #4142

OTX D-Fine Detection Algorithm Integration #4142

eugene123tw commented Dec 4, 2024 •

edited

Loading

kprokofi left a comment

kprokofi Dec 23, 2024

kprokofi Dec 23, 2024

kprokofi Dec 23, 2024

kprokofi Dec 23, 2024

kprokofi Dec 23, 2024

kprokofi Dec 23, 2024

kprokofi Dec 23, 2024

		return output.permute(0, 2, 1)


		class MSDeformableAttentionV2(nn.Module):


		PRETRAINED_ROOT: str = "https://github.com/Peterande/storage/releases/download/dfinev1.0/"

		PRETRAINED_WEIGHTS: dict[str, str] = {

		)


		def distance2bbox(points: Tensor, distance: Tensor, reg_scale: Tensor) -> Tensor:

		return box_convert(bboxes, in_fmt="xyxy", out_fmt="cxcywh")


		def deformable_attention_core_func_v2(

OTX D-Fine Detection Algorithm Integration #4142

Are you sure you want to change the base?

OTX D-Fine Detection Algorithm Integration #4142

Conversation

eugene123tw commented Dec 4, 2024 • edited Loading

Summary

OTX D-Fine Detection Algorithm Integration: https://github.com/Peterande/D-FINE

Next phase

How to test

Checklist

License

kprokofi left a comment

Choose a reason for hiding this comment

kprokofi Dec 23, 2024

Choose a reason for hiding this comment

kprokofi Dec 23, 2024

Choose a reason for hiding this comment

kprokofi Dec 23, 2024

Choose a reason for hiding this comment

kprokofi Dec 23, 2024

Choose a reason for hiding this comment

kprokofi Dec 23, 2024

Choose a reason for hiding this comment

kprokofi Dec 23, 2024

Choose a reason for hiding this comment

kprokofi Dec 23, 2024

Choose a reason for hiding this comment

eugene123tw commented Dec 4, 2024 •

edited

Loading