Add a stable diffusion xs + ControlNet pipeline

[lsb]

Pipeline description

I have a Stable Diffusion XS + ControlNet model that can compile to an ONNX model and run inference in WebGPU in a few hundred milliseconds in Chrome on a M1 Studio and in Chrome on a 4090.

In compiling this to an ONNX model, I’ve taken advantage of the fact that there’s only one step, so the scheduler phase and the time embedding compiled down to a constant. Further, the UNet + ControlNet + VAE model compresses down to 430MB for fast load times.

Is this sort of text to image synthesis model something that would be useful to add to a pipeline in Transformers.js? I’m open to collaborate on how best to refactor the monolithic ONNX artifact into something that makes sense for the existing code structure

Prerequisites

• The pipeline is supported in Transformers (i.e., listed here)
• The task is listed here

Additional information

Export script (requires a hot patched diffusers library!): https://github.com/lsb/sdxs-controlnet-sketch/blob/trunk/export-sdxs-512.py

Feel free to check out the repository that that’s in, or go to the linked blog post in that readme

Your contribution

Open to suggestions!