Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support for answerdotai/ModernBERT-base #457

Open
2 tasks done
mathcass opened this issue Dec 19, 2024 · 2 comments · May be fixed by #459
Open
2 tasks done

support for answerdotai/ModernBERT-base #457

mathcass opened this issue Dec 19, 2024 · 2 comments · May be fixed by #459

Comments

@mathcass
Copy link

mathcass commented Dec 19, 2024 •
edited
Loading

Model description

I tried running this on AWS SageMaker with the config,

    config = {
        "HF_MODEL_ID": "answerdotai/ModernBERT-base",
        "POOLING": "mean",
    }

but it failed with the following error message from the AWS console,

Error: Could not create backend
Caused by:
    Could not start backend: Model is not supported: unknown variant `modernbert`, expected one of `bert`, `xlm-roberta`, `camembert`, `roberta`, `distilbert`, `nomic_bert` at line 32 column 28

Open source status

  • The model implementation is available
  • The model weights are available

Provide useful links for the implementation

Released today, https://huggingface.co/blog/modernbert
@mhillebrand
Copy link

mhillebrand commented Dec 20, 2024 •
edited
Loading

I tried launching TEI-gRPC with Docker, using a fine-tuned ModernBERT model, and I got this error:

2024-12-20T20:05:51.464907Z  INFO text_embeddings_router: router/src/main.rs:175: Args { model_id: "/mod**/**nal", revision: None, tokenization_workers: None, dtype: Some(Float16), pooling: None, max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 4000, auto_truncate: false, default_prompt_name: None, default_prompt: None, hf_api_token: None, hostname: "81f1e6a2b3e2", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", cors_allow_origin: None }
thread 'main' panicked at /usr/src/router/src/lib.rs:134:62:
tokenizer.json not found. text-embeddings-inference only supports fast tokenizers: Error("data did not match any variant of untagged enum ModelWrapper", line: 251498, column: 1)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

And when I tried setting dtype to bfloat16:

error: invalid value 'bfloat16' for '--dtype <DTYPE>'
  [possible values: float16, float32]

tranformers 4.47.1
tokenizers 0.21.0
TEI 1.6.0

Upgrading to the latest main branch of transformers had the same problem.

@kozistr kozistr linked a pull request Dec 25, 2024 that will close this issue
Open
5 tasks
@kozistr
Copy link
Contributor

kozistr commented Dec 26, 2024

@mhillebrand @mathcass hi. I implemented the ModernBert model, and you can use it by building from the source based on #459! please feel free to leave a comment if you have an issue :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement the ModernBert model kozistr/text-embeddings-inference
3 participants
@mhillebrand @mathcass @kozistr