support for answerdotai/ModernBERT-base

[mathcass]

Model description

I tried running this on AWS SageMaker with the config,

    config = {
        "HF_MODEL_ID": "answerdotai/ModernBERT-base",
        "POOLING": "mean",
    }

but it failed with the following error message from the AWS console,

Error: Could not create backend
Caused by:
    Could not start backend: Model is not supported: unknown variant `modernbert`, expected one of `bert`, `xlm-roberta`, `camembert`, `roberta`, `distilbert`, `nomic_bert` at line 32 column 28

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Released today, https://huggingface.co/blog/modernbert

[mhillebrand]

I tried launching TEI-gRPC with Docker, using a fine-tuned ModernBERT model, and I got this error:

2024-12-20T20:05:51.464907Z  INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "/mod**/**nal", revision: None, tokenization_workers: None, dtype: Some(Float16), pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 4000, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "81f1e6a2b3e2", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }
thread 'main' panicked at /usr/src/router/src/lib.rs:134:62:
tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 251498, column: 1)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And when I tried setting dtype to bfloat16:

error: invalid value 'bfloat16' for '--dtype <DTYPE>'
  [possible values: float16, float32]

tranformers 4.47.1
tokenizers 0.21.0
TEI 1.6.0

Upgrading to the latest main branch of transformers had the same problem.

[kozistr]

@mhillebrand @mathcass hi. I implemented the ModernBert model, and you can use it by building from the source based on #459! please feel free to leave a comment if you have an issue :)