Depth conditioning is not working

[AtlantixJJ]

I am trying with the following example with a given depth prior.

The results are quite out of expectation.

On the other hand, if no depth conditioning is used, the results are somewhat closer:

Do you have a clue what's wrong with the depth conditioning part?

[pragyavaishanav]

I have similar issues.

To me it seems, it is assuming an underlying geometry of the model when we pass the front side image. Then it try to recreate geometry and then draw it on that geometry instead of taking the our custom depth maps into account properly. Look at my results. It was drawing two arms as if reading my depth and its OWN (concept of depth) depth and then try to work with both somehow but terribly failing.

[pragyavaishanav]

At least I know why you (@AtlantixJJ ) have issue. I was having similar problems as yours. I got rid of the scheduler that they are using. Once scheduler wasnt used, then it was drawing the input image on the depth maps (even though still weirdly).
I had added an issue related to the same previously here, #49

[eliphatfs]

The example script should have an output similar to this figure.
If you cannot reproduce the example script, please send me your diffusers, torch version and GPU setup, I guess there are compatibility issues instead of issues with the model.

[eliphatfs]

If you are good with the example but failing with your input, maybe you could tune the controlnet_scale parameter. In 2D SD controlnets this also needs to be tuned.
Another thing is that maybe the input is not matching the requirements of the depth condition. Please see if your scales or background is different to the example depth input.

[odusseys]

This is because of the scheduler used in the example code. Just remove it and the results will be correct.