Add attentive layer to Jepa #927
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
antoine-tran
changed the title
antoine-tran
changed the base branch from
main
to
tuan/support_explicit_init_fn
added 2 commits
December 20, 2024 13:46
|
I am outside for a couple errands, will review before EOD. |
cbalioglu
reviewed
Dec 20, 2024
src/fairseq2/recipes/jepa/models.py
Outdated
| def forward( | ||
| self, seqs: Tensor, padding_mask: PaddingMask | None | ||
| ) -> tuple[Tensor, PaddingMask | None]: | ||
| if self.encoder: |
There was a problem hiding this comment.
Are we sure that this forward implementation is accurate? In the reference implementation here, I see that encoder stack is applied after cross attention which is the reverse of what we have here.
There was a problem hiding this comment.
the internal code seems to deviate from this
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do? Please describe:
This PR follows up #889 to add the building blocks (models, builders, loader) for the finetuned JEPA encoder, plus testing scripts of the models in different downstream tasks.
Does your PR introduce any breaking changes? If yes, please list them:
List of all backwards-incompatible changes.
Check list: