Add support for Bumblebee functions?

[brainlid]

Bumblebee doesn't support a constrained output of only valid JSON.

In the early days of LangChain, they implemented an alternate approach for functions that was more of a hack, but worked well enough.

Investigate if this approach could work for bringing functions to Bumblebee models. It would still help if the model being run understood JSON, functions, etc.

[hwuethrich]

Hi @brainlid

I'd be very interested in this feature. Especially now that Mistral released their Mixtral-8x22B-Instruct-v0.1 model, which supports function calling! 🎉

Here is an example for the expected prompt template:
https://github.com/mistralai/mistral-common/blob/main/examples/tokenizer.ipynb

My company (@zebbra) might have some capacity to make this contribution in case you are interested? I assume it is just a matter of adding a new clause to https://github.com/brainlid/langchain/blob/main/lib/utils/chat_templates.ex, right? Or are there any other pieces missing from the puzzle?

Anyway, thanks for this library (and podcast! 🙂)

Edit: I guess ChatBumblebee would have to be able to parse the function calls as well

[hwuethrich]

Probably not as easy as I thought, because Mixtral is currently no supported by Bumblebee: elixir-nx/bumblebee#365

[marcnnn]

Hello,
I have added JSON tool use for Llama3.1 in #198
I will look into the other tool use methods and models as well if I am on the right track with my implementation, so please try it out!
And give feedback!