Is there somewhere I can find detailed documentation on NaVit?

[dempsey-ryan]

I'm having trouble understanding what the various parameters do, even after reading the source code.

Specifically, I'm wondering what group_max_seq_len does, and why it has non-deterministic results? For example:

v = NaViT(patch_size=60, **vit_args) # these are extremely large images

v(image_list, group_max_seq_len=1315)
tensor([[ 0.5456, -0.4548,  0.3367,  ...,  0.5904,  0.5517,  0.6039],
        [ 0.5456, -0.4548,  0.3367,  ...,  0.5904,  0.5517,  0.6039],
        [ 0.5456, -0.4548,  0.3367,  ...,  0.5904,  0.5517,  0.6039],
        ...,
        [ 0.5456, -0.4548,  0.3367,  ...,  0.5904,  0.5517,  0.6039],
        [ 0.5456, -0.4548,  0.3367,  ...,  0.5904,  0.5517,  0.6039],
        [ 0.5456, -0.4548,  0.3367,  ...,  0.5904,  0.5517,  0.6039]])
        
v(image_list, group_max_seq_len=229)
tensor([[ 0.2724, -0.8302,  0.4734,  ...,  0.7219,  0.6409,  0.4224],
        [ 0.5486, -0.4530,  0.3360,  ...,  0.5885,  0.5462,  0.6067],
        [ 0.2724, -0.8302,  0.4734,  ...,  0.7219,  0.6409,  0.4224],
        ...,
        [ 0.4754, -0.4497,  0.3625,  ...,  0.6052,  0.6225,  0.5106],
        [ 0.2724, -0.8302,  0.4734,  ...,  0.7219,  0.6409,  0.4224],
        [ 0.4645, -0.4711,  0.3736,  ...,  0.6147,  0.6285,  0.5033]])

For larger maximum sequence lengths, all the images have identical outputs. My problem is, I want deterministic results, therefore I want a constant max sequence length regardless of how the images are batched (kind of the whole reason I want to use NaViT). However, if I pick the maximum of the whole dataset, then you have the above (1315) result where every single image has identical logits.

If you can clarify how I decide on this parameter I would really appreciate it.