Got an out of range error when converting the fine-tuned model

[IcyXi]

I want to convert the fine-tuned Code Llama model file into a GGUF file, but got the following command:
python convert_hf_to_gguf.py /data/fine-tune/code_llama_fine_tune --outfile test.gguf --outtype q8_0
But I got an out an range error:

INFO:hf-to-gguf:Set meta model
INFO:hf-to-gguf:Set model parameters
INFO:hf-to-gguf:gguf: context length = 16384
INFO:hf-to-gguf:gguf: embedding length = 4096
INFO:hf-to-gguf:gguf: feed forward length = 11008
INFO:hf-to-gguf:gguf: head count = 32
INFO:hf-to-gguf:gguf: key-value head count = 32
INFO:hf-to-gguf:gguf: rope theta = 1000000
INFO:hf-to-gguf:gguf: rms norm epsilon = 1e-05
INFO:hf-to-gguf:gguf: file type = 7
INFO:hf-to-gguf:Set model tokenizer
WARNING:hf-to-gguf:ignore token 32003: id is out of range, max=31999
WARNING:hf-to-gguf:ignore token 32001: id is out of range, max=31999
WARNING:hf-to-gguf:ignore token 32000: id is out of range, max=31999
WARNING:hf-to-gguf:ignore token 32002: id is out of range, max=31999
Traceback (most recent call last):
  File "/data/llama.cpp/convert_hf_to_gguf.py", line 3688, in <module>
    main()
  File "/data/llama.cpp/convert_hf_to_gguf.py", line 3682, in main
    model_instance.write()
  File "/data/llama.cpp/convert_hf_to_gguf.py", line 401, in write
    self.prepare_metadata(vocab_only=False)
  File "/data/llama.cpp/convert_hf_to_gguf.py", line 394, in prepare_metadata
    self.set_vocab()
  File "/data/llama.cpp/convert_hf_to_gguf.py", line 1470, in set_vocab
    self._set_vocab_sentencepiece()
  File "/data/llama.cpp/convert_hf_to_gguf.py", line 693, in _set_vocab_sentencepiece
    tokens, scores, toktypes = self._create_vocab_sentencepiece()
  File "/data/llama.cpp/convert_hf_to_gguf.py", line 762, in _create_vocab_sentencepiece
    if toktypes[token_id] != SentencePieceTokenTypes.UNUSED:
IndexError: list index out of range

[Asherathe]

Same issue while using gguf-my-repo. Ever figure out what caused this?

[VolcanoJ25]

I have the same issue. any update?

[Asherathe]

In my case, the model I was trying to convert had a bad tokenizer due to a mergekit bug which has since been fixed. Redoing the merge with "embed_slerp: true" allowed it to be converted to gguf.

[FSSRepo]

This is because those models treat special tokens as regular tokens. There can only be 32,000 regular tokens, and if there are new ones, they must be included among the special tokens.

[VolcanoJ25]

Thanks. I'd like to understand this a bit better. In the model I'm using it lists this info:
tokenizer 32000
Special tokens: {'bos_token': '', 'eos_token': '', 'unk_token': '', 'pad_token': ''}
Special token IDs: [1, 2, 0]

so what should I be doing to avoid this error? I don't fully understand the math/logic here.