questions about the upcoming fp8 support

[Amin456789]

Hi!
@KohakuBlueleaf i see u r working on a fp8 in the changelogs. nice job and thanks, will it work with cpu and can we convert 1.5 models with it too?

[KohakuBlueleaf]

1.5 models are already supported for cuda(that's why it has -xl variation)

And for cpu I haven't checked
Since pytorch have autocast for cpu
Theoretically it could work(or I need to do lot of manual cast for it)

Basically the idea is param in fp8 but calculation in other precision
(But since for inference basically vram usage are from params, so it helps a lot)

[Amin456789]

thank u for answer!
please check and and make it cpu compatible as it will be a ram saver for us cpu users too, i have tested int8 onnx in the past [they are half the fp16] and they were so good for my cpu and ram. lots of people will benefit from this,
also, is there any tutorial how to convert the models to fp8?

thank u so much for ur hard work

[KohakuBlueleaf]

Just model.to(float8_e4m3fn)
(e4m3 is normally enough for all the usecase, but if you meet some problems that some param will overflow, use e5m2)

[Amin456789]

thank u!

[KohakuBlueleaf]

@Amin456789 I push a commit for CPU
and "it should work"
It can load model correctly, it can run sampling.
But since it is super slow on my machine (don't know if you have some optimization for CPU speed?)
I never run a full sampling loop for it

Need you guys to help to check if it work.

[KohakuBlueleaf]

I think I misunderstand your request
Your request should be fp8 for onnx right?

I don't know anything about onnx

[KohakuBlueleaf]

In my implementation
The fp8 is running in cpu and don't need any kind of conversion for the model

[Amin456789]

no i meant fp8 for torch models such as safetensors or ckpt.
so i can use a fp16 model here without converting to fp8? i mean we convert 32 to 16 to get the half size, i thought we could shrink it even more by going 8

[KohakuBlueleaf]

Wut, just convert it.
What I'm doing is for computing not store the model

[Amin456789]

oh now i get it :D