Use sequence length axis in tensor trim #723
Conversation
There was a problem hiding this comment.
should we add such models with different sequence length dimension ID to GHA CI validation ?
There was a problem hiding this comment.
Would we keep stateful mode after CB merge? If yes, then I think it make sense to add tests.
There was a problem hiding this comment.
Yes, CB is already merged and we keep stateful mode.
There was a problem hiding this comment.
Tests can be added separately
| const std::map<std::string, size_t> model_type_to_seq_len_axis{ | ||
| {"chatglm", 0}, | ||
| {"llama", 2}, | ||
| }; |
There was a problem hiding this comment.
@ilya-lavrenov , @Wovchena , This approach is not reliable, chatglm3 and chatglm2 have seq_len_axis=0 but glm-4-9b-chat has seq_len_axis=2. All of them have same model_type='chatglm'.
I'll try to detect seq_len_axis by comparing kv tensor shape dimensions with generated tokens length.
|
Removed from merge queue due to: #723 (comment) |
as-suvorov
changed the title
trimm_tensor