We already have #9119 open with the same topic. If you are on metal, MLX just merged support for it on mlx_llm.

MLX just merged support for it on mlx_llm.

It looks like they just merged support for 3.5. Does that include MoE?
riccardomusmeci/mlx-llm@1b6fe71

@linkage001

@gardner

MLX just merged support for it on mlx_llm.

It looks like they just merged support for 3.5. Does that include MoE? riccardomusmeci/mlx-llm@1b6fe71

@linkage001

@gardner yes, it includes MoE. It's running at 35tk/s on my M1 MacBook pro. ml-explore/mlx-examples#946

I am using Ubuntu 23.10 with 2xEPYC 9654 12 channel DDR5.

thank you but where gguf file?

thanks for the site but it fails to convert

https://github.com/foldl/chatllm.cpp

| Supported Models | Download Quantized Models |

What's New:

2024-08-28: Phi-3.5 Mini & MoE

Inference of a bunch of models from less than 1B to more than 300B, for real-time chatting with RAG on your computer (CPU), pure C++ implementation based on @ggerganov's ggml.

| Supported Models | Download Quantized Models |

What's New:

2024-08-28: Phi-3.5 Mini & MoE

not run on colab

There is a PR in transformers. Maybe this is a requirement for llama.cpp to support the conversion? huggingface/transformers#33363.

This issue was closed because it has been inactive for 14 days since being marked as stale.

plox reopen mr bot

The PR in the transformers repo has been merged and is featured in release v4.46.0.

Here is the documentation for how to add a new model to llama.cpp: https://github.com/ggerganov/llama.cpp/blob/master/docs/development/HOWTO-add-model.md

I wish I knew enough about C++, maybe I try it for lols not that I'll get far 😅

I don't know anything about AI/ML either outside of basics lol