The training loss is so weird

[kkkoaoa]

Description

df_ep0100_rgb.mp4

df_ep0100_depth.mp4

Steps to Reproduce

1.Training statements：python main.py --text "a hotdog" --workspace trial2 -O2
2.It's the first training session after the previous steps
3.Its training loss is particularly strange, either large in the tens of thousands or zero

Expected Behavior

normal loss not ab normal loss

Environment

pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118

[Lpigge]

issue65 have same problem？maybe you can check it and get something.