OpenVINO(TM) Toolkit support

[avbelova]

Add a support for OpenVINO(TM) - toolkit for optimizing and deploying AI inference. This pull request allows to accelerate EasyOCR on Intel(R) platforms: Intel(R) CPU, Intel(R) Processor Graphics or Intel(R) Discrete Graphics.

[raymondlo84]

Shall we test and merge this?

[jonashaag]

Seems to work and give a 30 % performance boost on Intel CPU

[raymondlo84]

Seems to work and give a 30 % performance boost on Intel CPU

works with 2024.3? and let me put a few folks validating it :)

[raymondlo84]

Can you please share the instructions to install or build this setup?

[raymondlo84]

so I did git clone and checkout this
then
python3 -m venv venv
source venv/bin/activate
python setup.py install

then, here is an error

Installed /home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/openvino_telemetry-2024.1.0-py3.10.egg
error: numpy 2.0.1 is installed but numpy<2.0.0,>=1.16.6 is required by {'openvino'}

import easyocr
reader = easyocr.Reader(['ch_sim','en'], ov_cpu) # this needs to run only once to load the model into memory
result = reader.readtext('chinese.jpg')
print(result)               

When I run the sample, here we got this

(venv) raymondlo84@4thSPR-Flex:/home2/raymondlo84/EasyOCR/EasyOCR/examples$ python test.py 
Traceback (most recent call last):
  File "/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/torch-2.4.0-py3.10-linux-x86_64.egg/torch/__init__.py", line 226, in _load_global_deps
    ctypes.CDLL(global_deps_lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcudart.so.12: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home2/raymondlo84/EasyOCR/EasyOCR/examples/test.py", line 1, in <module>
    import easyocr
  File "/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr-1.7.1-py3.10.egg/easyocr/__init__.py", line 1, in <module>
    from .easyocr import Reader
  File "/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr-1.7.1-py3.10.egg/easyocr/easyocr.py", line 3, in <module>
    from .recognition import get_recognizer, get_text
  File "/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr-1.7.1-py3.10.egg/easyocr/recognition.py", line 2, in <module>
    import torch
  File "/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/torch-2.4.0-py3.10-linux-x86_64.egg/torch/__init__.py", line 289, in <module>
    _load_global_deps()
  File "/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/torch-2.4.0-py3.10-linux-x86_64.egg/torch/__init__.py", line 247, in _load_global_deps
    _preload_cuda_deps(lib_folder, lib_name)
  File "/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/torch-2.4.0-py3.10-linux-x86_64.egg/torch/__init__.py", line 170, in _preload_cuda_deps
    ctypes.CDLL(lib_path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libnvJitLink.so.12: cannot open shared object file: No such file or directory

How do I build this without using CUDA? I think it maybe related to that?

[avbelova]

Seems to work and give a 30 % performance boost on Intel CPU

Glad that you were able to successfully run EasyOCR with OpenVINO backend. Which exact Intel CPU did you use for the test? On the latest generations of Intel(R)Core and Intel(R) Xeon CPUs performance boost should be bigger.

[avbelova]

Can you please share the instructions to install or build this setup?

$ python3 -m venv env
$ source env/bin/activate
$ pip install git+https://github.com/avbelova/EasyOCR.git

[raymondlo84]

Looks like it is install lots of CUDA stuffs :|

Using legacy 'setup.py install' for easyocr, since package 'wheel' is not installed.
Installing collected packages: python-bidi, pyclipper, openvino-telemetry, ninja, mpmath, typing-extensions, sympy, PyYAML, Pillow, packaging, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, MarkupSafe, fsspec, filelock, triton, tifffile, Shapely, scipy, openvino, opencv-python-headless, nvidia-cusparse-cu12, nvidia-cudnn-cu12, lazy-loader, jinja2, imageio, scikit-image, nvidia-cusolver-cu12, torch, torchvision, easyocr

shall we make a cleaner version with OV?

(venv) raymondlo84@4thSPR-Flex:~/EasyOCR/EasyOCR/examples$ python test.py
Neither CUDA nor MPS are available - defaulting to CPU. Note: This module is much faster with a GPU.
/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr/detection.py:89: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
net.load_state_dict(copyStateDict(torch.load(trained_model, map_location=device)))
/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr/recognition.py:189: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state_dict = torch.load(model_path, map_location=device)
[([[86, 80], [134, 80], [134, 128], [86, 128]], '西', 0.6629598563364745), ([[189, 75], [469, 75], [469, 165], [189, 165]], '愚园路', 0.9619015323198089), ([[517, 81], [565, 81], [565, 123], [517, 123]], '东', 0.9935730580773452), ([[78, 126], [136, 126], [136, 156], [78, 156]], '315', 0.9999928421498292), ([[514, 126], [574, 126], [574, 156], [514, 156]], '309', 0.9999620772373462), ([[79, 173], [125, 173], [125, 213], [79, 213]], 'W', 0.2336988002212701), ([[226, 170], [414, 170], [414, 220], [226, 220]], 'Yuyuan Rd。', 0.8949639433306373), ([[529, 173], [569, 173], [569, 213], [529, 213]], 'E', 0.5179029432256108)]

With no OV works.

[raymondlo84]

(venv) raymondlo84@4thSPR-Flex:~/EasyOCR/EasyOCR/examples$ python test.py 
/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr/detection.py:98: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  net.load_state_dict(copyStateDict(torch.load(trained_model, map_location='cpu')))
Text detection model is running with OpenVINO on Intel CPU
/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr/recognition.py:198: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state_dict = torch.load(model_path, map_location="cpu")
Text recognition model is running with OpenVINO on Intel  CPU
[([[86, 80], [134, 80], [134, 128], [86, 128]], '西', 0.8170299794057065), ([[189, 75], [469, 75], [469, 165], [189, 165]], '愚园路', 0.9587661652434012), ([[517, 81], [565, 81], [565, 123], [517, 123]], '东', 0.9935737710310519), ([[78, 126], [136, 126], [136, 156], [78, 156]], '315', 0.9999945627862629), ([[514, 126], [574, 126], [574, 156], [514, 156]], '309', 0.9999705427375312), ([[79, 173], [125, 173], [125, 213], [79, 213]], 'W', 0.27741190629603807), ([[226, 171], [414, 171], [414, 220], [226, 220]], 'Yuyuan Rd。', 0.759156469717586), ([[529, 173], [569, 173], [569, 213], [529, 213]], 'E', 0.5343126463518644)]
(venv) raymondlo84@4thSPR-Flex:~/EasyOCR/EasyOCR/examples$ 

And with OV works too :)

[raymondlo84]

AND GPU works (I got Intel Flex!!)

(venv) raymondlo84@4thSPR-Flex:~/EasyOCR/EasyOCR/examples$ python test.py 
/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr/detection.py:98: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  net.load_state_dict(copyStateDict(torch.load(trained_model, map_location='cpu')))
Text detection model is running with OpenVINO on Intel GPU
/home2/raymondlo84/EasyOCR/EasyOCR/venv/lib/python3.10/site-packages/easyocr/recognition.py:198: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state_dict = torch.load(model_path, map_location="cpu")
Text recognition model is running with OpenVINO on Intel  GPU
[([[86, 80], [134, 80], [134, 128], [86, 128]], '西', 0.8091345858697565), ([[189, 75], [469, 75], [469, 165], [189, 165]], '愚园路', 0.9576007553265619), ([[517, 81], [565, 81], [565, 123], [517, 123]], '东', 0.9942458436325161), ([[78, 126], [136, 126], [136, 156], [78, 156]], '315', 0.9999933927534448), ([[514, 126], [574, 126], [574, 156], [514, 156]], '309', 0.9999633160904321), ([[79, 173], [125, 173], [125, 213], [79, 213]], 'W', 0.3235391723110048), ([[226, 170], [414, 170], [414, 220], [226, 220]], 'Yuyuan Rd。', 0.8545725454434623), ([[529, 173], [569, 173], [569, 213], [529, 213]], 'E', 0.5599389980489811)]
(venv) raymondlo84@4thSPR-Flex:~/EasyOCR/EasyOCR/examples$ ```

[jonashaag]

Seems to work and give a 30 % performance boost on Intel CPU

Glad that you were able to successfully run EasyOCR with OpenVINO backend. Which exact Intel CPU did you use for the test? On the latest generations of Intel(R)Core and Intel(R) Xeon CPUs performance boost should be bigger.

Intel Xeon-E 2274G, this was the only CPU available to me. I didn't try the GPU though

[raymondlo84]

Seems to work and give a 30 % performance boost on Intel CPU

Glad that you were able to successfully run EasyOCR with OpenVINO backend. Which exact Intel CPU did you use for the test? On the latest generations of Intel(R)Core and Intel(R) Xeon CPUs performance boost should be bigger.

Intel Xeon-E 2274G, this was the only CPU available to me. I didn't try the GPU though

I tested it and It would work on intel laptop's iGPU too. (Tried it on Intel Core Ultra)

[jonashaag]

What's the speedups you are seeing?

[avbelova]

What's the speedups you are seeing?

It depends on several factors like input picture size, language, CPU/GPU model, etc. On Intel® Core™ Ultra CPU the speedup was ~3.5x comparing with Pytorch CPU, on 5th Gen Intel® Xeon® Scalable from 3x up to 5.8x comparing with Pytorch CPU.