Model Degradation During Fine-Tuning on Mesh Representation with Depth and Normal Maps

[SBlumenstock1]

I'm experiencing a degradation in model performance when fine-tuning with mesh representation (depth and normal maps). The model initially works well when fine-tuned on the NeRF representation but begins to degrade after a few steps when fine-tuning with mesh-based training. This issue affects depth, normal maps, and the overall 3D reconstruction quality, as seen in the attached training logs.

It seems like the loss for geometry-related aspects (depth, normals) might not be contributing adequately, or the SDF regularization is too aggressive, leading to collapsing geometry. I have tested to remove the loss_reg and the error has disappeared, so the problem is within the calculation of the flexicubes regularization loss.

Any insights or suggestions would be greatly appreciated!

[feiqiu-cyber]

I'm experiencing a degradation in model performance when fine-tuning with mesh representation (depth and normal maps). The model initially works well when fine-tuned on the NeRF representation but begins to degrade after a few steps when fine-tuning with mesh-based training. This issue affects depth, normal maps, and the overall 3D reconstruction quality, as seen in the attached training logs.

It seems like the loss for geometry-related aspects (depth, normals) might not be contributing adequately, or the SDF regularization is too aggressive, leading to collapsing geometry. I have tested to remove the loss_reg and the error has disappeared, so the problem is within the calculation of the flexicubes regularization loss.

Any insights or suggestions would be greatly appreciated!

Hello, I met the same problem as you, have you solved it?