Paper | ArXiv | Project Page | Supp_material | Video
old_assets.mp4
assets_1.mp4
assets_appearance.mp4
assets.mp4
Please pull the latest code to improve performance!!
- (2023.10.17)
- Add a FAQ about the performance of the official code and the reproduced version of threestudio, i.e. Q8.
- (2023.10.13)
- Add a FAQ about the problem of over-saturation and over-smoothing appearance, i.e. Q7.
- (2023.09.01)
- Note: adding negative prompts can improve the visual quality in appearance modeling. See the latest Gallery for more details.
- (2023.08.30)
- Add two FAQs.
- (2023.08.29)
- Skip some unnecessary material prediction processes in geometry modeling, which slightly decreases the training time.
- (2023.08.22)
- Note: For situations where the requirement for the number of mesh faces is not high, I recommend using a DMTet resolution of 128 to achieve a balance between quality and generation speed. When using a resolution of 128 or less, the generation speed will be significantly faster.
- (2023.08.20)
- Note: different versions of stable diffusion seem to have different performance. In terms of generating avatars, version 1.5 performs better than version 2.1.
- (2023.08.11)
- Fix the bug about the unsmooth geometry in some cases.
- (2023.08.06)
- Add a tip to improve the realism of appearance.
- (2023.08.05)
- Fix the bug about the same seed between different gpus, which may not accelerate convergence.
- Add a part for directly performing UV padding in the code, so there is no need for additional operations in the blender.
- (2023.07.26)
- Fix the bug about using the tangent space normal.
Q1: About the use of normal and mask images as the input of stable diffusion model and analysis
Answer: Our initial hypothesis is that normal and mask images, representing local and silhouette information of shapes respectively, can benefit geometry learning. Additionally, we observed that the value range of the normal map is normalized to (-1, 1), which aligns with the data range required for latent space diffusion. Our empirical studies validate this hypothesis. Further support for our hypothesis comes from the presence of normal images in the LAION-5B dataset used for training Stable Diffusion (see Website for retrieval of normal data in LAION-5B). Therefore, the normal data is not considered an out-of-distribution (OOD) input for stable diffusion. To handle rough and coarse geometry in the early stage of learning, we directly utilize concatenated 64
Q2: Hypothesis-verification analysis of the disentangled representation
Answer: Previous methods (e.g., DreamFusion and Magic3D) couple the geometry and appearance generation together, following NeRF. Our adoption of the disentangled representation is mainly motivated by the difference of problem nature for generating surface geometry and appearance. In fact, when dealing with finer recovery of surface geometry from multi-view images, methods (e.g., VolSDF, nvdiffrec, etc) that explicitly take the surface modeling into account triumph; our disentangled representation enjoys the benefit similar to these methods. The disentangled representation also enables us to include the BRDF material representation in the appearance modeling, achieving better photo-realistic rendering by the BRDF physical prior.
Q3: Can Fantasia3D directly fine-tune the mesh given by the user?
Answer: Yes, it can. Fantasia3D can receive any mesh given by the user and fine-tune it using our method of user-guided generation. It can also naturally interface with the 3D generative method like shape-e and point-e. In a word, Fantasia3D can generate highly detailed and high-fidelity 3D content based on either the low-quality mesh given by the users or the ellipsoid.
Q4: What do you think is the reason why it could not replicate the same result with 4 or fewer GPUs when using the official configs?
Answer: The official configs are usually used under 8 GPUs. The sampling algorithm proposed in the supplementary materials contributes to global consistency in appearance and geometry modeling, and it depends on a large batch size. When using fewer GPUs, the overall batch size is significantly smaller, which can result in the inability to replicate the same results in the official configs. One possible solution is to manually increase the batch size in the configs.
Q5: How do strategy 0, 1 and 2 in appearance modeling come up?
Answer: The strategy weight is a hyperparameter. The parameter, i.g.
where
Q6: How can I make the generated results more diverse?
Answer: Unlike Nerf-based volume rendering, directly using surface rendering to generate 3D assets can achieve diversity. You only need to change the parameters in the configuration file to achieve that. For example, different "sdf_init_shape_scale", "translation_y", "camera_random_jitter", "fovy_range", "negative_text" etc can bring you different results.
Q7: Can Fantasia3D generate photorealistic appearance without oversaturation and over-smooth?
Answer: Yes, it can. Just Original SDS loss combined with negative prompt and strategy 2 proposed in Fantasia3D is enough to address the oversaturated and over-smoothing problem. The appearance of the DMTet-based gallery in Sweetdreamer is generated by the code of appearance modeling in Fantasia3D. You can see that all the results are highly detailed and do have not the problem of oversaturation and over-smooth. I think the key is the disentangled representation and the adoption of the negative prompt and strategy 2. The recommended negative prompt is "shadow, oversaturated, low quality, unrealistic". The recommended positive prompt is "a DSLR photo of ...".
Q8: What is the difference between the official code and the reproduced version of threestudio?
Answer: For geometry modeling, official code has stronger generalization ability, more stable training process, and smoother geometry. For appearance modeling, the official code does not have the problem of oversaturation and oversmoothing and has the SOTA text-to-texture performance, as mentioned in Q7.
Considering that parameter tuning may require some experience, what kind of object do you want me to generate? Please speak freely in the issue area. I will take some time to implement some requirements and update the corresponding configuration files for your convenience in reproducing.
Firstly, upload the videos converted from gifs using the Website, including the geometry or appearance, to the Gallery. Write down the text to generate the object, the performance, the resolution of the tetrahedron for geometry modeling, and the strategy adopted for appearance modeling.
Subsequently, upload the configuration file under the directory of configs. If you will upload the file about the user-guided generation, the guided mesh should also be uploaded under the directory of data. The naming rule of the file is as follows.
For the file of zero-shot geometry modeling:
{The key word of the text}_geometry_zero_shot_{the number of gpu}_gpu.jsonFor the file of user-guided geometry modeling:
{The key word of the text}_geometry_user_guided_{the number of gpu}_gpu.jsonFor the file of appearance modeling:
{The key word of the text}_appearance_strategy{the strategy adopted}_{the number of gpu}_gpu.json.- System requirement: Ubuntu20.04
- Tested GPUs: RTX3090, RTX4090, A100, V100
We provide two choices to install the environment.
-
(Option 1) Use the file requirements.txt to install all packages one by one. It may fail since the complexity of some packages.
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html pip install -r requirements.txt
-
(Option 2) Use the docker image to deploy the environment in the Ubuntu system quickly.
docker pull registry.cn-guangzhou.aliyuncs.com/baopin/fantasia3d:1.0
(optional) Due to the Internet Network Delay, the package of xformers was not installed in this docker image. Install it by hand to accelerate the training during geometry and appearance modeling after you create a docker container using the above docker image.
pip install git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
After the successful deployment of the environment, clone the repository of Fantasia3D and get started.
git clone https://github.com/Gorilla-Lab-SCUT/Fantasia3D.git
cd Fantasia3DAll the results in the paper were generated using 8 3090 GPUs. We cannot guarantee that fewer than 8 GPUs can achieve the same effect.
- zero-shot generation
# Multi-GPU training
...
# Geometry modeling using 8 GPU
python3 -m torch.distributed.launch --nproc_per_node=8 train.py --config configs/car_geometry.json
# Geometry modeling using 4 GPU
python3 -m torch.distributed.launch --nproc_per_node=4 train.py --config configs/car_geometry.json
# Appearance modeling using 8 GPU
python3 -m torch.distributed.launch --nproc_per_node=8 train.py --config configs/car_appearance_strategy0.json
# Appearance modeling using 4 GPU
python3 -m torch.distributed.launch --nproc_per_node=4 train.py --config configs/car_appearance_strategy0.json
...
# Single GPU training (Only test on the pineapple).
# Geometry modeling. It takes about 15 minutes on 3090 GPU.
python3 train.py --config configs/pineapple_geometry_single_gpu.json
# Appearance modeling. It takes about 15 minutes on 3090 GPU.
python3 train.py --config configs/pineapple_appearance_strategy0_single_gpu.json- user-guided generation
# Multi-GPU training
...
# Geometry modeling using 8 GPU
python3 -m torch.distributed.launch --nproc_per_node=8 train.py --config configs/Gundam_geometry.json
# Geometry modeling using 4 GPU
python3 -m torch.distributed.launch --nproc_per_node=4 train.py --config configs/Gundam_geometry.json
# Appearance modeling using 8 GPU
python3 -m torch.distributed.launch --nproc_per_node=8 train.py --config configs/Gundam_appearance.json
# Appearance modeling using 4 GPU
python3 -m torch.distributed.launch --nproc_per_node=4 train.py --config configs/Gundam_appearance.json
...
# Single GPU training
# Geometry modeling (untested)
python3 train.py --config configs/Gundam_geometry.json
# Appearance modeling (untested)
python3 train.py --config configs/Gundam_appearance.json-
(both) Train longer. Training longer may help with the finer details. You can train longer by setting the parameter "iter".
-
(both) Larger batch size. A larger batch size can help with the faster convergence. Corresponding parameter is "batch".
-
(both) Try different seeds. Different seeds can bring diverse results.
-
(both) Scale the object. Increasing the proportion of initialized objects in the FOV = 45 screens can reinforce the quality of both the geometry and appearance modeling. For geometry modeling, it can attain more local geometric details. For appearance modeling, this method can reduce the probability of saturated or strange colors appearing, as it reduces the proportion of background colors in the image. We found that if the proportion of background color is too high, it can easily lead to saturation and strange colors.
-
(geometry modeling) Provide a proportional prior of the target shape. You can scale the default sphere with a radius of 1 to an ellipsoid. For instance, make the radius of the ellipsoid on the z-axis larger if you want to generate "A car made out of cheese".
"mode": "geometry_modeling", "sdf_init_shape": "ellipsoid", "sdf_init_shape_scale": [0.56, 0.56, 0.84]
There is a situation where ellipsoid cannot provide a proportional prior, such as the generation of an animal. In this case, using ellipsoid initialization can easily cause the generated animal to have multiple feet. Run the following command to examine:
python3 -m torch.distributed.launch --nproc_per_node=8 train.py --config configs/elephant_geometry_fail_multi_face.json
Instead, you can use the sketch shape of a quadruped as a proportional prior to generating any animal shape you want.
python3 -m torch.distributed.launch --nproc_per_node=8 train.py --config configs/elephant_geometry_succeed.json
In other situations, such as the generation of the human-like body, a human sketch shape can be used.
python3 -m torch.distributed.launch --nproc_per_node=8 train.py --config configs/Gundam_geometry.json
-
(geometry modeling) Increae the number of iterations in the early phase. The early phase is very crucial to create a coarse and correct shape. The late phase just focuses on attaining finer geometry details so there will be no significant changes in the overall shape. Increase the number of the parameter "coarse_iter" if you find that the contour of the geometric shape does not match the text description.
-
(geometry modeling) Use larger resolution of the tetrahedron. For situations where the requirement for the number of mesh faces is not high, I recommend using a DMTet resolution of 128 to achieve a balance between quality and generation speed. When using a resolution of 128 or less, the generation speed will be significantly faster. A larger resolution can bring more details in the local geometry. You can easily change the resolution by modifying the value of the parameter "dmtet_grid" to 128 or 256. Note that if you find that the mesh quickly disappears or disperses when using 256 resolution, decrease the guidance weight of SDS loss from default 100 to 50. In my experience, a single GPU is suitable for using a resolution of 128 instead of 256. If you want to obtain a high-detail model at 256 resolution, multi-GPU training is necessary. In addition, the effect of multiple GPUs is much better than a single GPU for objects with obvious directionality, such as human head statues. BTW, using the gradient accumulation technique for a single GPU may achieve the effect of multiple GPUs, but I haven't tested it yet.
-
(geometry modeling) Use different range of time step in the early phase. We usually use the time steps range [0.02,0.5] in the early phase. But in some cases where you want to "grow" more parts based on the initialized shape, it may fail to generate all parts. For instance, the text "An astronaut riding a horse", may fail to "grow" the part of the astronaut using the range [0.02, 0.5] since the fact that low time steps have little contribution to significant deformation. To address this problem, we recommend you use a high range, such as [0.4, 0.6]. You can try different ranges and publish your findings in the issue.
-
(geometry modeling) Rotate the object. Rotating the object according to the actual situation can alleviate janus-problem or help the network in mode-seeking. For example, when generating a human head statue, rotate the initialized ellipsoid around the x-axis by some angle to match the situation where the back of the person's head has some curvature.
-
(geometry modeling) Fine-tune the input mesh. Under the task of user-guided generation, if you are satisfied with the silhouette of the input mesh and just want to increase the details of the geometry, set the parameter "coarse_iter" to 400. This setting would directly enter the late phase of geometry modeling which reinforces the local geometric details of the input shape.
-
(appearance modeling) Use different strategy. We offer three strategy (0 or 1 or 2) to optimize the appearance by setting the parameter "sds_weight_strategy". For strategy 0, there will be stronger light and shadow changes, representing a more realistic final appearance. For strategy 1 or 2, the final appearance will be smoother and more comfortable. If the target appearance is too simple, such as "a highly detailed stone bust of Theodoros Kolokotronis", "A standing elephant", and "Michelangelo style statue of dog reading news on a cellphone", using strategy 0 may lead to an oversaturated appearance and strange color. In this case, strategy 1 or 2 can generate more natural color than strategy 0.
strategy 0 can be used as follow.
"sds_weight_strategy": 0, "early_time_step_range": [0.02, 0.98], "late_time_step_range": [0.02, 0.5]
or
"sds_weight_strategy": 0, "early_time_step_range": [0.02, 0.98], "late_time_step_range": [0.02, 0.98]
strategy 1 can be used as follow:
"sds_weight_strategy": 1, "early_time_step_range": [0.02, 0.98], "late_time_step_range": [0.02, 0.7]
or
"sds_weight_strategy": 1, "early_time_step_range": [0.02, 0.98], "late_time_step_range": [0.02, 0.98]
strategy 2 can be used as follow:
"sds_weight_strategy": 2, "early_time_step_range": [0.02, 0.98], "late_time_step_range": [0.02, 0.98]
-
(appearance modeling) Use different HDR environment maps. Learning the PBR materials is an ill-posed problem. If materials and lighting are learned together, it will increase the difficulty of learning. So We use the fixed HDR light to optimize the appearance. We noticed that HDR maps with uniform brightness distribution, such as cloudy days, are conducive to the uniformity of appearance colors. Some uneven brightness distribution may produce more realistic results (untested).
-
(appearance modeling) Use some additional prompts. Using some additional prompts, such as "a DSLR photo of {your text}, black background" can improve the realism of appearance in most cases (but not all).
-
(appearance modeling) Add negative prompt. Setting appropriate negative prompts can improve the visual quality in appearance modeling.
You can download and watch some demos' training process in Google drive For more demos see here
girl_geometry.mp4
girl_appearance.mp4
baby_bunny_geometry.mp4
bunny_appearance_strategy0.mp4
dog_appearance_stratege0.mp4
dog_geometry.mp4
sandcastle_geometry.mp4
sandcastle_appearacne_strategy0.mp4
sandcastle_appearance_strategy1.mp4
sandcastle_appearance_strategy2.mp4
Gundam_geometry.mp4
Gundam_appearance.mp4
mug_geometry.mp4
mug_appearance.mp4
Einstein.mp4
pineapple_geometry.mp4
pineapple_appearance.mp4
car_geometry.mp4
car_appearance_strategy0.mp4
car_appearance_strategy1.mp4
elephant_geometry.mp4
- Release the code. (2023.06.15)
- Support the gradient accumulation technique for single GPU training.
- Support the VSD loss proposed by ProlificDreamer.
@InProceedings{chen2023fantasia3d,
author={Chen, Rui and Chen, Yongwei and Jiao, Ningxin and Jia, Kui},
title={Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month={October},
year={2023},
pages={22246-22256}
}