Pose Proposal Network Tested + Pre-Trained Models => Google Drive + D…

…ocumentation Update (#302) * Fix PPN post-processing * Fix PPN post-processing: re * Clean PPN header * Remove useless scripts * More details on TensorRT building * Doc Comment on PPN * Models => Google Drive | Update Documents * Format C++ codes Update CI Update Install Scripts Update Install Scripts Update Install Scripts * Fix FAKE
tensorlayer · Aug 17, 2020 · 33c168a · 33c168a
1 parent d247583
commit 33c168a
Show file tree

Hide file tree

Showing 38 changed files with 395 additions and 388 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -7,16 +7,29 @@ jobs:
 
     # https://help.github.com/en/articles/virtual-environments-for-github-actions#supported-virtual-environments
     runs-on: ubuntu-18.04
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8]
 
     steps:
-    - uses: actions/checkout@v1
-    - run: sudo apt-get install libopencv-dev libgflags-dev # dependencies
-    - run: sh scripts/download-test-data.sh
-    - run: sh scripts/download-tinyvgg-model.sh
-    - run: sh scripts/download-openpose-thin-model.sh
-    - run: sh scripts/download-openpose-res50-model.sh
-    - run: sh scripts/download-openpose-coco-model.sh
-    - run: cmake . -DBUILD_TESTS=1 -DBUILD_FAKE=1 -DBUILD_EXAMPLES=1 -DBUILD_LIB=1 -DBUILD_USER_CODES=0 -DEXECUTABLE_OUTPUT_PATH=./bin
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Initialize Python Env
+      run: python3 -m pip install --upgrade pip
+    - name: Install System Dependencies
+      run: sudo apt-get install libopencv-dev libgflags-dev # dependencies
+    - name: Check download scripts.
+      run: |
+        sh scripts/download-test-data.sh
+        sh scripts/download-tinyvgg-model.sh
+        sh scripts/download-openpose-thin-model.sh
+        sh scripts/download-openpose-res50-model.sh
+        sh scripts/download-openpose-coco-model.sh
+    - name: Build Project(NO GPU)
+      run: cmake . -DBUILD_TESTS=1 -DBUILD_FAKE=1 -DBUILD_EXAMPLES=1 -DBUILD_LIB=1 -DBUILD_USER_CODES=0 -DEXECUTABLE_OUTPUT_PATH=./bin
     - run: cmake --build . --config Release
 
     # - run: ctest -C Release

diff --git a/.gitignore b/.gitignore
@@ -44,6 +44,7 @@ venv
 _build
 docs/make.bat
 examples/user_codes/*.cpp
+debug.*
 
 !docs/Makefile
 !docs/markdown/images/*
diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@
 
 HyperPose is a library for building human pose estimation systems that can efficiently operate in the wild.
 
-> **Note**: We are in the process of migrating our APIs from 1.0 to 2.0. The migration is expected to finish by July 2020.
+> **News**: The PoseProposal inference model is released! See the HyperPose models on [Google Drive](https://drive.google.com/drive/folders/1w9EjMkrjxOmMw3Rf6fXXkiv_ge7M99jR?usp=sharing).
 
 ## Features
 
@@ -19,14 +19,14 @@ You can install HyperPose and learn its APIs through [Documentation](https://hyp
 
 ## Example
 
-We provide an example to show human pose estimation achieved by HyperPose. You need to install CUDA Toolkit 10+, TensorRT 7+, OpenCV 3.2+ and gFlags (cmake version), and enable C++ 17 support. Once the prerequisite are ready, run the following script:
+We provide an example to show human pose estimation achieved by HyperPose. You need to install CUDA Toolkit 10+, TensorRT 7+, OpenCV 3.2+ and gFlags (cmake version), and enable C++ 17 support. Once the prerequisite are met, run the following script:
 
 ```bash
-sudo apt -y install git cmake build-essential subversion curl libgflags-dev # libopencv-dev # [optional] 
+sudo apt -y install git cmake build-essential subversion libgflags-dev libopencv-dev
 sh scripts/download-test-data.sh         # Install data for examples.
 sh scripts/download-tinyvgg-model.sh     # Install tiny-vgg model.
 mkdir build && cd build
-cmake .. -DCMAKE_BUILD_TYPE=RELEASE && make -j$(nproc) # Build library && examples.
+cmake .. -DCMAKE_BUILD_TYPE=RELEASE && make -j # Build library && examples.
 ./example.operator_api_batched_images_paf    # The ouput images will be in the build folder.
 ```
 

diff --git a/docs/markdown/design/design.md b/docs/markdown/design/design.md
@@ -47,7 +47,7 @@ int main() {
     using namespace hyperpose;
 
     const cv::Size network_resolution{384, 256};
-    const dnn::uff uff_model{ "../data/models/hao28-600000-256x384.uff", "image", {"outputs/conf", "outputs/paf"} };
+    const dnn::uff uff_model{ "../data/models/TinyVGG-V1-HW=256x384.uff", "image", {"outputs/conf", "outputs/paf"} };
 
     // * Input video.
     auto capture = cv::VideoCapture("../data/media/video.avi");
@@ -106,7 +106,7 @@ int main() {
     using namespace hyperpose;
 
     const cv::Size network_resolution{384, 256};
-    const dnn::uff uff_model{ "../data/models/hao28-600000-256x384.uff", "image", {"outputs/conf", "outputs/paf"} };
+    const dnn::uff uff_model{ "../data/models/TinyVGG-V1-HW=256x384.uff", "image", {"outputs/conf", "outputs/paf"} };
 
     // * Input video.
     auto capture = cv::VideoCapture("../data/media/video.avi");

diff --git a/docs/markdown/install/prediction.md b/docs/markdown/install/prediction.md
@@ -6,10 +6,13 @@
 * CMake 3.5+ 
 * Third-Party
     * OpenCV3.2+.
-    * [CUDA 10](https://developer.nvidia.com/cuda-downloads), [TensorRT 7](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt_304/tensorrt-install-guide/index.html).
+    * [CUDA 10.2](https://developer.nvidia.com/cuda-downloads), [CuDNN 7.6.5](https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html), [TensorRT 7.0](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html). (For Linux users, [Debian Installation](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-debian) is highly recommended.)
     * gFlags(optional, for examples/tests)
 
-> Older versions of the packages may also work but not tested.
+> Other versions of the packages may also work but not tested.
+
+> Different TensorRT version requires specific CUDA and CuDNN version. For specific CUDA and CuDNN requirements of TensorRT7, please refer to [this](https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#platform-matrix).
+> Also, for Ubuntu 18.04 users, this [3rd party blog](https://ddkang.github.io/2020/01/02/installing-tensorrt.html) may help you. 
 
 ## Build On Ubuntu 18.04
 
@@ -18,7 +21,7 @@
 sudo apt -y install cmake libopencv-dev  
 # You may also install OpenCV from source to get best performance.
 
-# >>> Install CUDA/TensorRT
+# >>> Install CUDA/CuDNN/TensorRT
 
 # >>> Build gFlags(Optional) from source. Install it if you want to run the examples.
 wget https://github.com/gflags/gflags/archive/v2.2.2.zip
@@ -32,7 +35,7 @@ sudo make install
 git clone https://github.com/tensorlayer/hyperpose.git
 cd hyperpose
 mkdir build && cd build
-cmake .. -DCMAKE_BUILD_TYPE=RELEASE && make -j$(nproc)
+cmake .. -DCMAKE_BUILD_TYPE=Release && make -j
 ```
 
 ## Build with User Codes

diff --git a/docs/markdown/performance/prediction.md b/docs/markdown/performance/prediction.md
@@ -13,8 +13,6 @@
 > **Environment**: [email protected], GPU@1070Ti, CPU@i7(12 logic cores). 
 >
 > **Tested Video Source**: Crazy Updown Funk(resolution@640x360, frame_count@7458, source@[YouTube](https://www.youtube.com/watch?v=2DiQUX11YaY))
->
-> **Availability**: All model above are available [here](https://github.com/tensorlayer/pretrained-models/tree/master/models/hyperpose). 
 
 > OpenPose performance is not tested with batch processing as it seems not to be implemented. (see [here](https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/100)) 
 

diff --git a/docs/markdown/performance/supports.md b/docs/markdown/performance/supports.md
@@ -16,11 +16,8 @@
 ### Supported Post-Processing Methods
 
 - Part Association Field(PAF)
-- Pose Proposal Networks(Coming Soon)
+- Pose Proposal Networks
 
 ### Released Prediction Models
 
-- [Tiny VGG](https://github.com/tensorlayer/pretrained-models/blob/master/models/hyperpose/hao28-600000-256x384.uff)
-- [OpenPose-COCO](https://github.com/tensorlayer/pretrained-models/blob/master/models/hyperpose/openpose_coco.onnx)
-- [OpenPose-Thin](https://github.com/tensorlayer/pretrained-models/blob/master/models/hyperpose/openpose_thin.onnx)
-- [ResNet18(for PAF)](https://github.com/tensorlayer/pretrained-models/blob/master/models/hyperpose/lopps_resnet50.onnx)
+We released the models on [Google Drive](TinyVGG-V1-HW=256x384.uff). `.onnx` and `.uff` files are for inference.
diff --git a/docs/markdown/quick_start/prediction.md b/docs/markdown/quick_start/prediction.md
@@ -30,9 +30,10 @@ sh scripts/download-openpose-thin-model.sh      # ~20  MB
 sh scripts/download-tinyvgg-model.sh            # ~30  MB
 sh scripts/download-openpose-res50-model.sh     # ~45  MB
 sh scripts/download-openpose-coco-model.sh      # ~200 MB
+sh scripts/download-ppn-res50-model.sh          # ~50  MB (PoseProposal Algorithm)
 ```
 
-> You can download them manually to `${HyperPose}/data/models/` via [LINK](https://github.com/tensorlayer/pretrained-models/tree/master/models/hyperpose) **if the network is not working**.
+> You can download them manually to `${HyperPose}/data/models/` via [LINK](https://drive.google.com/drive/folders/1w9EjMkrjxOmMw3Rf6fXXkiv_ge7M99jR?usp=sharing) **if the network is not working**.
 
 ## Predict a sequence of images
 
@@ -46,17 +47,23 @@ sh scripts/download-openpose-coco-model.sh      # ~200 MB
 # Take images in ../data/media as a big batch and do prediction.
 
 ./example.operator_api_batched_images_paf
-# The same as: `./example.operator_api_batched_images_paf --model_file ../data/models/hao28-600000-256x384.uff --input_folder ../data/media --input_width 384 --input_height 256`
+# The same as: `./example.operator_api_batched_images_paf --model_file ../data/models/TinyVGG-V1-HW=256x384.uff --input_folder ../data/media --input_width 384 --input_height 256`
 ```
 
 The output images will be in the build folder.
 
 ### Using a precise model
 
 ```bash
-./example.operator_api_batched_images_paf --model_file ../data/models/openpose_thin.onnx --input_width 432 --input_height 368 
+./example.operator_api_batched_images_paf --model_file ../data/models/openpose-thin-V2-HW=368x432.onnx --input_width 432 --input_height 368 
 
-./example.operator_api_batched_images_paf --model_file ../data/models/openpose_coco.onnx --input_width 656 --input_height 368 
+./example.operator_api_batched_images_paf --model_file ../data/models/openpose-coco-V2-HW=368x656.onnx --input_width 656 --input_height 368 
+```
+
+### Use PoseProposal model
+
+```bash
+./example.operator_api_batched_images_pose_proposal --model_file ../data/models/ppn-resnet50-V2-HW=384x384.onnx --input_width 368 --input_height 368
 ```
 
 ### Convert models into TensorRT Engine Protobuf format

diff --git a/docs/markdown/tutorial/faq.md b/docs/markdown/tutorial/faq.md
@@ -24,7 +24,7 @@ Refer to [here](https://www.learnopencv.com/tag/install/).
 
 Download them manually:
 
-- All prediction models are available [here](https://github.com/tensorlayer/pretrained-models/tree/master/models/hyperpose).
+- All prediction models are available on [Google Drive](https://drive.google.com/drive/folders/1w9EjMkrjxOmMw3Rf6fXXkiv_ge7M99jR?usp=sharing).
 - The test data are taken from the [OpenPose Project](https://github.com/CMU-Perceptual-Computing-Lab/openpose/tree/master/examples/media).
 
 ## Training 
@@ -34,7 +34,7 @@ Download them manually:
 ### TensorRT Error?
 
 - See the `tensorrt.log`. (it contains more informations about logging and is located in where you execute the binary)
-- You may meet `ERROR: Tensor image cannot be both input and output` when using the `hao28-600000-256x384.uff` model. And just ignore it.
+- You may meet `ERROR: Tensor image cannot be both input and output` when using the `TinyVGG-V1-HW=256x384.uff` model. And just ignore it.
 
 ### Performance?
 

diff --git a/docs/markdown/tutorial/prediction.md b/docs/markdown/tutorial/prediction.md
@@ -133,7 +133,7 @@ int main() {
     using namespace hyperpose;
 
     const cv::Size network_resolution{384, 256};
-    const dnn::uff uff_model{ "../data/models/hao28-600000-256x384.uff", "image", {"outputs/conf", "outputs/paf"} };
+    const dnn::uff uff_model{ "../data/models/TinyVGG-V1-HW=256x384.uff", "image", {"outputs/conf", "outputs/paf"} };
 
     // * Input video.
     auto capture = cv::VideoCapture("../data/media/video.avi");

diff --git a/examples/gen_serialized_engine.example.cpp b/examples/gen_serialized_engine.example.cpp
@@ -4,7 +4,7 @@
 #include <string_view>
 
 // Model flags
-DEFINE_string(model_file, "../data/models/hao28-600000-256x384.uff", "Path to uff model.");
+DEFINE_string(model_file, "../data/models/TinyVGG-V1-HW=256x384.uff", "Path to uff model.");
 
 DEFINE_bool(logging, false, "Print the logging information or not.");
 DEFINE_string(input_name, "image", "The input node name of your model file. (for Uff model, input/output name tags required)");

diff --git a/examples/operator_api_batched_images_paf.example.cpp b/examples/operator_api_batched_images_paf.example.cpp
@@ -4,7 +4,7 @@
 #include <string_view>
 
 // Model flags
-DEFINE_string(model_file, "../data/models/hao28-600000-256x384.uff", "Path to uff model.");
+DEFINE_string(model_file, "../data/models/TinyVGG-V1-HW=256x384.uff", "Path to uff model.");
 
 DEFINE_bool(logging, false, "Print the logging information or not.");
 

diff --git a/examples/operator_api_batched_images_pose_proposal.example.cpp b/examples/operator_api_batched_images_pose_proposal.example.cpp
@@ -4,7 +4,7 @@
 #include <string_view>
 
 // Model flags
-DEFINE_string(model_file, "../data/models/ppn.onnx", "Path to uff model.");
+DEFINE_string(model_file, "../data/models/ppn-resnet50-V2-HW=384x384.onnx", "Path to uff model.");
 DEFINE_int32(input_width, 384, "Width of input image.");
 DEFINE_int32(input_height, 384, "Height of input image.");
 

diff --git a/examples/operator_api_imshow_paf.example.cpp b/examples/operator_api_imshow_paf.example.cpp
@@ -3,7 +3,7 @@
 #include <hyperpose/hyperpose.hpp>
 
 // Model flags
-DEFINE_string(model_file, "../data/models/hao28-600000-256x384.uff", "Path to uff model.");
+DEFINE_string(model_file, "../data/models/TinyVGG-V1-HW=256x384.uff", "Path to uff model.");
 DEFINE_string(input_name, "image", "The input node name of your uff model file.");
 DEFINE_string(output_name_list, "outputs/conf,outputs/paf", "The output node names(maybe more than one) of your uff model file.");
 

diff --git a/examples/operator_api_imshow_pose_proposal.example.cpp b/examples/operator_api_imshow_pose_proposal.example.cpp
@@ -0,0 +1,82 @@
+#include "utils.hpp"
+#include <gflags/gflags.h>
+#include <hyperpose/hyperpose.hpp>
+
+// Model flags
+DEFINE_string(model_file, "../data/models/ppn-resnet50-V2-HW=384x384.onnx", "Path to uff model.");
+DEFINE_int32(input_width, 384, "Width of input image.");
+DEFINE_int32(input_height, 384, "Height of input image.");
+
+DEFINE_bool(logging, false, "Print the logging information or not.");
+
+DEFINE_string(input_video, "../data/media/video.avi", "The input video path.");
+DEFINE_bool(camera, false, "Using the camera as input video.");
+
+int main(int argc, char** argv)
+{
+    gflags::ParseCommandLineFlags(&argc, &argv, true);
+
+    // * Input video.
+    auto capture = FLAGS_camera ? cv::VideoCapture(0) : cv::VideoCapture(FLAGS_input_video);
+    if (!capture.isOpened())
+        example_log() << "Cannot open cv::VideoCapture.";
+
+    // * Create TensorRT engine.
+    namespace hp = hyperpose;
+    if (FLAGS_logging)
+        hp::enable_logging();
+
+    auto engine = [&] {
+        using namespace hp::dnn;
+        constexpr std::string_view onnx_suffix = ".onnx";
+        constexpr std::string_view uff_suffix = ".uff";
+
+        if (std::equal(onnx_suffix.crbegin(), onnx_suffix.crend(), FLAGS_model_file.crbegin()))
+            return tensorrt(onnx{ FLAGS_model_file }, { FLAGS_input_width, FLAGS_input_height }, 1);
+
+        example_log() << "Your model file's suffix is not [.onnx | .uff]. Your model file path: " << FLAGS_model_file;
+        example_log() << "Trying to be viewed as a serialized TensorRT model.";
+
+        return tensorrt(tensorrt_serialized{ FLAGS_model_file }, { FLAGS_input_width, FLAGS_input_height }, 1);
+    }();
+
+    // * post-processing: Using Pose Proposal.
+    hp::parser::pose_proposal parser{ engine.input_size() };
+
+    using clk_t = std::chrono::high_resolution_clock;
+
+    example_log() << "Inference Started. Use ESC to quit.";
+
+    while (capture.isOpened()) {
+
+        cv::Mat mat;
+        capture >> mat;
+        if (mat.empty()) {
+            example_log() << "Got empty cv::Mat";
+            break;
+        }
+
+        auto beg = clk_t::now();
+
+        {
+            // * TensorRT Inference.
+            auto feature_maps = engine.inference({ mat });
+
+            // * Post-Processing.
+            auto poses = parser.process(feature_maps.front());
+
+            for (auto&& pose : poses)
+                hp::draw_human(mat, pose);
+        }
+
+        double fps = 1000. / std::chrono::duration<double, std::milli>(clk_t::now() - beg).count();
+
+        cv::putText(mat, "FPS: " + std::to_string(fps), { 10, 10 }, cv::FONT_HERSHEY_SIMPLEX, 0.5, { 0, 255, 0 }, 2);
+        cv::imshow("HyperPose Prediction", mat);
+
+        if (cv::waitKey(1) == 27)
+            break;
+    }
+
+    example_log() << "Inference Done!";
+}
diff --git a/examples/operator_api_video_paf.example.cpp b/examples/operator_api_video_paf.example.cpp
@@ -3,7 +3,7 @@
 #include <hyperpose/hyperpose.hpp>
 
 // Model flags
-DEFINE_string(model_file, "../data/models/hao28-600000-256x384.uff", "Path to uff model.");
+DEFINE_string(model_file, "../data/models/TinyVGG-V1-HW=256x384.uff", "Path to uff model.");
 DEFINE_string(input_name, "image", "The input node name of your uff model file.");
 DEFINE_string(output_name_list, "outputs/conf,outputs/paf", "The output node names(maybe more than one) of your uff model file.");
 

diff --git a/examples/operator_api_video_pose_proposal.example.cpp b/examples/operator_api_video_pose_proposal.example.cpp
@@ -3,7 +3,7 @@
 #include <hyperpose/hyperpose.hpp>
 
 // Model flags
-DEFINE_string(model_file, "../data/models/ppn.onnx", "Path to uff model.");
+DEFINE_string(model_file, "../data/models/ppn-resnet50-V2-HW=384x384.onnx", "Path to uff model.");
 DEFINE_int32(input_width, 384, "Width of input image.");
 DEFINE_int32(input_height, 384, "Height of input image.");
 DEFINE_int32(max_batch_size, 8, "Max batch size for inference engine to execute.");
@@ -52,7 +52,7 @@ int main(int argc, char** argv)
     }();
 
     // * post-processing: Using Pose Proposal
-    hp::parser::pose_proposal parser{ engine.input_size(), 0.05 };
+    hp::parser::pose_proposal parser{ engine.input_size() };
 
     using clk_t = std::chrono::high_resolution_clock;
 

diff --git a/examples/stream_api_video_paf.example.cpp b/examples/stream_api_video_paf.example.cpp
@@ -3,7 +3,7 @@
 #include <hyperpose/hyperpose.hpp>
 #include <opencv2/opencv.hpp>
 
-DEFINE_string(model_file, "../data/models/hao28-600000-256x384.uff",
+DEFINE_string(model_file, "../data/models/TinyVGG-V1-HW=256x384.uff",
     "Path to uff model.");
 DEFINE_string(input_name, "image", "The input node name of your uff model file.");
 DEFINE_string(output_name_list, "outputs/conf,outputs/paf", "The output node names(maybe more than one) of your uff model file.");

diff --git a/examples/stream_api_video_pose_proposal.example.cpp b/examples/stream_api_video_pose_proposal.example.cpp
@@ -3,7 +3,7 @@
 #include <hyperpose/hyperpose.hpp>
 #include <opencv2/opencv.hpp>
 
-DEFINE_string(model_file, "../data/models/ppn.onnx", "Path to uff model.");
+DEFINE_string(model_file, "../data/models/ppn-resnet50-V2-HW=384x384.onnx", "Path to uff model.");
 DEFINE_int32(input_width, 384, "Width of input image.");
 DEFINE_int32(input_height, 384, "Height of input image.");
 DEFINE_int32(max_batch_size, 8, "Max batch size for inference engine to execute.");

diff --git a/examples/tutorial/minimum_operator_api_video.example.cpp b/examples/tutorial/minimum_operator_api_video.example.cpp
@@ -5,7 +5,7 @@ int main()
     using namespace hyperpose;
 
     const cv::Size network_resolution{ 384, 256 };
-    const dnn::uff uff_model{ "../data/models/hao28-600000-256x384.uff", "image", { "outputs/conf", "outputs/paf" } };
+    const dnn::uff uff_model{ "../data/models/TinyVGG-V1-HW=256x384.uff", "image", { "outputs/conf", "outputs/paf" } };
 
     // * Input video.
     auto capture = cv::VideoCapture("../data/media/video.avi");

diff --git a/examples/tutorial/minimum_stream_api_video.example.cpp b/examples/tutorial/minimum_stream_api_video.example.cpp
@@ -5,7 +5,7 @@ int main()
     using namespace hyperpose;
 
     const cv::Size network_resolution{ 384, 256 };
-    const dnn::uff uff_model{ "../data/models/hao28-600000-256x384.uff", "image", { "outputs/conf", "outputs/paf" } };
+    const dnn::uff uff_model{ "../data/models/TinyVGG-V1-HW=256x384.uff", "image", { "outputs/conf", "outputs/paf" } };
 
     // * Input video.
     auto capture = cv::VideoCapture("../data/media/video.avi");