Clarification for behavior of en_core_web_lg with doc.tensor

[dennlinger]

Using the en_core_web_lg model, I was comparing the output from doc.tensor, and doc.vector.
Until recently, I was expecting doc.tensor to be a matrix representation of the stacked word vectors (i.e., [token.vector for token in doc]).

As it turns out, doc.tensor is in fact similar in the dimension to the doc.tensor produced by en_core_web_sm or en_core_web_md (although the matrix entries do not match), which is not the dimension of the distinct token.vectors, and doc.vector.

I was wondering whether this definition should be clarified in the Doc documentation page. I have not spotted any comment on the differences (or similarities) of the two properties. This is also especially relevant when considering the calculation of .similarity(), where it is not immediately clear from the docs whether this is based on doc.tensor or doc.vector.

Furthermore, I could not find any comment on the specific computation of the tensors, so maybe a paragraph on this could also help clarify this issue?

Which page or section is this issue related to?

https://github.com/explosion/spaCy/blob/master/website/docs/api/doc.md

[svlandeg]

The way I understand it, the tensors hold information related to the predictions of the statistical models (such as the tagger or parser), while the vectors represent semantic similarity. The similiarity() method uses the vectors in principle. However, the vector attributes have a sort of fall-back in case they are not set, and then they do refer to the tensor, such as in the case of using the en_core_web_sm model, cf the docs:

To make them compact and fast, spaCy’s small models (all packages that end in sm) don’t ship with word vectors, and only include context-sensitive tensors. This means you can still use the similarity() methods to compare documents, spans and tokens – but the result won’t be as good, and individual tokens won’t have any vectors assigned.

But I think you're right that this could be clarified in the docs more.

[priya55612]

Doc.vector is a real-valued meaning representation. Defaults to an average of the token vectors. Got it from the link https://spacy.io/api/doc

So, they are taking the average. If doc.tensor.shape = (7,768) then doc.vector.shape = (768,) nothing taking an average of all the 7 rows with 768 dimensions giving one row with 768 dimensions.
Please correct if I am wrong!

[svlandeg]

This section has been further expanded for the v3 docs: https://nightly.spacy.io/usage/linguistic-features#vectors-similarity.

So I think we can close this one? If you have further suggestions for clarifications in the docs, feel free to open a PR :-)