Windows10 CUDA 10.1 VS2019 Detectron2 Build - nvcc fail

[elnazsn1988]

Referencing facebookresearch/detectron2#232 , I followed to facebookresearch/detectron2#9 and found this solution. I cloned this version of detectron2 and implemented the ROI changes in pytorch, running the setup.py command in x64 Native Tools Command prompt for VS2019, with the cd set as the folder detectron2 has been cloned to.

I have hit error:

nvcc fatal : Could not set up the environment for Microsoft Visual Studio using 'C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.23.28105/bin/HostX86/x64/../../../../../../../VC/Auxiliary/Build/vcvars64.bat'
error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\nvcc.exe' failed with exit status 1

complete error log:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\nvcc.exe -c C:\projects\pytorch\pytorch\detectron2\detectron2\layers\csrc\box_iou_rotated\box_iou_rotated_cuda.cu -o build\temp.win-amd64-3.7\Release\projects\pytorch\pytorch\detectron2\detectron2\layers\csrc\box_iou_rotated\box_iou_rotated_cuda.obj -IC:\projects\pytorch\pytorch\detectron2\detectron2\layers\csrc -IC:\Users\aesnj\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\include -IC:\Users\aesnj\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\aesnj\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\include\TH -IC:\Users\aesnj\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include" -IC:\Users\aesnj\AppData\Local\Programs\Python\Python37\include -IC:\Users\aesnj\AppData\Local\Programs\Python\Python37\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" -Xcompiler /EHsc -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75
nvcc fatal : Could not set up the environment for Microsoft Visual Studio using 'C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.23.28105/bin/HostX86/x64/../../../../../../../VC/Auxiliary/Build/vcvars64.bat'
error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\nvcc.exe' failed with exit status 1

[Zy1013921842]

@elnazsn1988 Hello, friend. Have you solved this problem? I just get the same error. If you solved it, please tell me how to make it work, thanks so much!

[zzw0225]

okay,i got the same problem..

[elnazsn1988]

@elnazsn1988 Hello, friend. Have you solved this problem? I just get the same error. If you solved it, please tell me how to make it work, thanks so much!

okay,i got the same problem..

@zzw0225 @Zy1013921842 I posted what worked for me in one of the links:

" so what worked for me, was using the conansherry detectron2, doing the changes here to every possible version of the documents saved on my system: #237 (comment) , changing the ROI files (rotated and not cu) and not sure if this means anything, but also cloning in parallel in Conda. I wasnt able to actually get a complete run in Conda, but somehow now it "seems" to be working."

The changes made to files :
several files must be changed by manually.
file1:
{your evn path}\Lib\site-packages\torch\include\torch\csrc\jit\argument_spec.h
example:
{C:\Miniconda3\envs\py36}\Lib\site-packages\torch\include\torch\csrc\jit\argument_spec.h(190)
static constexpr size_t DEPTH_LIMIT = 128;
change to -->
static const size_t DEPTH_LIMIT = 128;
file2:
{your evn path}\Lib\site-packages\torch\include\pybind11\cast.h
example:
{C:\Miniconda3\envs\py36}\Lib\site-packages\torch\include\pybind11\cast.h(1449)
explicit operator type&() { return *(this->value); }
change to -->
explicit operator type&() { return ((type)this->value); }

This has also been mentioned here :
facebookresearch/detectron2#9 (comment)

I also changed two files in number 12 of facebookresearch/maskrcnn-benchmark#1042 , namely the ROIAligned rotated cu file in sysWoW64. As Im not sure what I changed here, I will try and attach my SysWow64/detectron2/detectron2/layers/csrc folder bellow and you can compare to see if same as yours in ROIAligned.

I also conda installed as opposed to pip installed after doing all this and it worked. Which one worked, I did not figure out unfortunaley, as I did not try to reproduce. I did all this having uninstalled and reinstalling everything to ensure correct versions was being accessed.

I would suggest trying to implement changes mentioned step by step and seeing where it fixes issue, but make sure you have same windows, VS and cuda versions as the post.

[elnazsn1988]

csrc.zip

[solarflarefx]

@elnazsn1988 Is changing the cu files for ROIAligned and ROIAlignedRotated still required in the newer versions of pytorch?