Fix panic in DecodeStream::step due to incorrect index usage #1699
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When calling
DecodeStream::step
multiple times, it eventually panics with attempt to subtract with overflow in the following lines of code:tokenizers/tokenizers/src/tokenizer/mod.rs
Lines 1108 to 1109 in 24d29f4
The panic can be easily reproduced, and I have added a test case to demonstrate the issue.
Upon inspecting the code, I found that the shrinking of the token buffer references
read_index
instead ofprefix_index
. This PR corrects the issue by using the correct index.However, this change makes
read_index
unused, so I am not entirely certain if it aligns with the intended logic of the original implementation. Please let me know if further adjustments or clarifications are needed, or if there is additional context regarding the intended use ofread_index
.