Entity Recognition from Search Queries

[atalnarayan]

Hi,

I am working on a search application, and want to extract relevant named entities from the search query. I am thinking of using spacy's Parts of Speech tagger for the same.

Using Parts of Speech tagger, I can find topics/entities using contiguous chunks of important PoS tags. I can also find relationship between those terms using adposition terms.

E.g. Consider the query -- "Gaming culture in India".

Using spacy PoS tagger, we get the following (token, Pos tag) mappings:
[('gaming', 'VERB'), ('culture', 'NOUN'), ('in', 'ADP'), ('India', 'PROPN')].

Considering Nouns and Verbs as important identifiers, we extract phrases as: gaming culture, India. Moreover, India is a proper noun, so it is likely a place, or a proper noun like a chapter of some book. Therefore, we can say that gaming culture is a topic, while India is an important entity.

My question is:

1. Is my idea useful? Can you think of cases where it might fail?
2. Can I improve this by using spacy's dependency parser? If yes, how?
3. Are there other ways to do entity extraction on queries? If yes, Can you share the relevant implementations with me?

[kadarakos]

Hey atalnarayan,

In general the approach you are taking seems to be on the right track, but your question is a bit general for a discussion here. Let me point you to some relevant material:

1. Finding video game titles with sense2vec: https://www.youtube.com/watch?v=EoYHbUHr0fM
2. Detailed example about using the entity ruler to find museum names: https://www.youtube.com/watch?v=Ds18bQAzygo.
3. Rather than the EntityRuler we recommend using the SpanRuler in the future: https://spacy.io/api/spanruler
4. Using dependency tree for extracting information: https://www.youtube.com/watch?v=BoyLPiXXEYA&t=429s.
5. For more in-depth information about entity extraction I recommend this Chapter: https://web.stanford.edu/~jurafsky/slp3/8.pdf
6. For practical examples for machine learning based named entity recognition with spacy you can checkout the relevant projects here: https://github.com/explosion/projects.

[atalnarayan]

Thanks @kadarakos. Your response was helpful and gave me enough resources to further finetune my approach. Really appreciate your help.