Massive LLM2Vec testing

[BradKML]

I managed to make somewhat of a compilation of models to test for different benchmarks, and I think MTEB is unique in that it it not meant for LLM proper but instead the "embeddings" implicit to LLM. LiveBench/LiveBench#95

Some questions to consider:

• Will fine-tuned models and merges be better or worse than foundation models?
• Are there benefits to franken-merge models like the ones noted by SakanaAI?
• Are bigger models with more dimension better than smaller models?
• When will purpose-made embeddings be better than LLM hacks (or vice versa)?

[KennethEnevoldsen]

Hi @BradKML,

1. It depends on how you define fine-tuning. Generally models adapting for document embedding and retrieval are notably better than default embeddings from a model
2. Potentially, but we haven't seen these for embeddings
3. Generally larger models outperform smaller models, but the scaling laws are not as clear for embedding with multiple smaller models performing quite well.
4. Depends on what you mean by LLM hacks - see 1)

[BradKML]

For 1, There are fine-tuned models that are more geared towards hacking for results of certain benchmarks (which leads to degraded performance in others), while others that are trained generally with more and higher quality data for enhanced performance.
For 2, Would love experiments to see how it turns out (the following 2 groups are examples) https://huggingface.co/freewheelin https://huggingface.co/arcee-ai
For 4, I was referring to the difference between purpose-built embeddings like nomic, bge, and mxbai, compared to LLM2Vec of SOTA LLM models.

[KennethEnevoldsen]

4. sota LLM (decoder) models generally do not perform well for embedding tasks and typically require some sort of adaption an example of such an adaption is the LLM2Vec. You could argue that LLM2Vec is unsupervised (enable attention, train with masked language modeling followed by unsupervised SimCSE) to which I would agree. However, the SOTA LLM2Vec models have been further trained in a supervised fashion (similar to bge, mxbai). We have not seen unsupervised text encoders (as far as I am aware) perform better when creating semantic embeddings for search, classification etc.

[BradKML]

So can it be said that supervised embeddings and LLM2Vec are somewhat comparable, even though the latter has been trained for longer and encompasses a wider range of data (and only lacking the "vec" part in post)?