Load controlnet with diffusers

[YutingXiao]

Thanks for this excellent work! But when I use diffusers to load Controlnet:

controlnet = SD3ControlNetModel.from_pretrained("stabilityai/stable-diffusion-3.5-controlnets-depth", torch_dtype=torch.float16)

I got error:

Traceback (most recent call last):
  File "/home/xxxx/SyncMVD-main/run_experiment_batch_sd35.py", line 22, in <module>
    controlnet = SD3ControlNetModel.from_pretrained("stabilityai/stable-diffusion-3.5-controlnets-depth", torch_dtype=torch.float16)
  File "/home/xxxx/miniconda3/envs/syncmvd/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/xxxx/miniconda3/envs/syncmvd/lib/python3.10/site-packages/diffusers/models/modeling_utils.py", line 730, in from_pretrained
    model = cls.from_config(config, **unused_kwargs)
  File "/home/xxxx/miniconda3/envs/syncmvd/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 260, in from_config
    model = cls(**init_dict)
  File "/home/xxxx/miniconda3/envs/syncmvd/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 653, in inner_init
    init(self, *args, **init_kwargs)
  File "/home/xxxx/miniconda3/envs/syncmvd/lib/python3.10/site-packages/diffusers/models/controlnet_sd3.py", line 75, in __init__
    self.context_embedder = nn.Linear(joint_attention_dim, caption_projection_dim)
  File "/home/xxxx/miniconda3/envs/syncmvd/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 96, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
TypeError: empty(): argument 'size' failed to unpack the object at pos 2 with error "type must be tuple of ints,but got NoneType"

It seems the version of diffusers should be "0.32.0.dev0" which is shown in the config.json file, but the newest version of diffusers is "0.31.0" when I use pip to install. So, do you know how to solve this??

[Eric-Hao]

Hi @YutingXiao, you can resolve the issue by pip install git+https://github.com/huggingface/diffusers to install the 0.32.0.dev0 version. This command will install the latest development build directly from GitHub.