Metal support?

[kchro3]

Hi, awesome work on this project!

I'm building some Swift apps using llama.cpp, and I'd love to try getting clip.cpp running on my app too.

I'm curious if you're going to support running clip.cpp on Metal like llama.cpp?

[monatis]

Hi @kchro3, thanks for reaching out! It needs Conv2D implementation on Metal. I'm currently implementing LLaVA with llama.cpp and clip.cpp combined (initial release in a few days). The CLIP part will run on CPU in the first place, but then I'll implement Conv2D on Metal to support end-to-end Metal inference for multimodal models.

[kchro3]

wow, that's awesome. i'm looking forward to the release!

[kchro3]

where can i follow your progress btw? will it be in this repo or are you going to make a new one

[monatis]

WIP PR for the initial release of large multimodal model inference is in another repo. Metal implementation of Conv2D will require another pr to the main GGML repo after this PR is merged and released.

[kchro3]

awesome work, @monatis ! cheering you on

[yysu-888]

Hi, awesome work on this project!

I'm building some Swift apps using llama.cpp, and I'd love to try getting clip.cpp running on my app too.

I'm curious if you're going to support running clip.cpp on Metal like llama.cpp?
welcome to my project, support all openai clip and ofa-sys chinese clip model,at the same time support metal gpu
and f16,q8_0 quantization
https://github.com/yysu-888/clip.cpp