GPT2Attention() class with _attn() method when add_cross_attention=True and therefore is_cross_attention=True.

[CHLEE-Leo]

Feature request

Model description

It seems like GPT2Attention() class allows_attn() method with causal_mask only when is_cross_attention=False, but not when is_cross_attention=True.

It would be more productive if GPT2Attention() supports _attn() method with causal_mask even with is_cross_attention=True.

Motivation

When developing EncoderDecoderModel where the encoder is ViTModel and the decoder isGPT2Model, the current GPT2Model class does not support causal_mask if add_cross_attention=True and therefore is_cross_attention=True.

Your contribution

No contribution.