not sure if this can be integrated

[gsgoldma]

but it looks like it can fine tune a lot of stuff a bit better than control net, at least in different ways, if im not mistaken

https://github.com/TencentARC/T2I-Adapter

[Mikubill]

Seems like the structure is similar but adapted on unet's encode phrase.

Will try if we could integrate it.

[matrix4767]

Apparently this supports dual model guidance, that is 2 models at once, which would be superb.
But this has the Apache license, so is it even doable?

[gsgoldma]

cant believe you already integrated some of it! way to go!