How to filter the documents based on metadata in opensearch

[prasannasaikommineni]

I am trying to filter based on id in metadata I tried in below ways but I am not getting the filtered data

docsearch = OpenSearchVectorSearch.from_documents(
docs, CustomEmbeddings,
http_auth=awsauth,
index_name="index_name",
timeout=300,
opensearch_url="opensearch_url",
connection_class=RequestsHttpConnection,
engine="faiss"
)

docsearch.similarity_search(query, k=10 ,search_type="script_scoring",
pre_filter={"match": {"id": 3}})
and also tried as below
docsearch.similarity_search(query, k=10 ,search_type="script_scoring",
pre_filter={"match": {"metadata.id": 3}})

can someone please help me with the format for filter object

[dosubot[bot]]

🤖

Hi @prasannasaikommineni! 😊 I'm Dosu, a friendly bot here to assist you with bugs, queries, or contributing to the repo. Don't worry, a human maintainer will be with you soon, but in the meantime, I'm here to help!

Based on the context provided, it seems like you're trying to filter documents based on the id field in the metadata. The pre_filter parameter in the similarity_search method is used to pre-filter documents before identifying nearest neighbors. If not provided, it defaults to a MATCH_ALL_QUERY, which means all documents are considered for the search.

However, the way you're trying to use the pre_filter parameter might not be correct. The pre_filter parameter expects a valid OpenSearch query. The match query you're trying to use is a full-text query and might not work as expected with numerical fields like id.

Instead, you could try using a term query, which is more suitable for exact value matches. Here's how you can do it:

docsearch.similarity_search(query, k=10 ,search_type="script_scoring",
                                                  pre_filter={"term": {"metadata.id": 3}})

Please note that the field name (metadata.id in this case) should match exactly with the field name in your OpenSearch index. If the field name is different, you'll need to adjust it accordingly.

Also, please ensure that the id field is not analyzed in your OpenSearch index. The term query works on the exact terms that are stored in the inverted index and would not work correctly with analyzed fields.

If you're still facing issues, could you please provide more details about the structure of your documents and the exact error message you're getting? This would help in identifying the issue more accurately.

Sources

• libs/langchain/langchain/vectorstores/opensearch_vector_search.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[prasannasaikommineni]

How can I filter a metadata field with text values .I want to prefilter the data bsed on country field
my documents Array is like below [Document(page_content='This is a cow . This is a cow .\n\nThis is a cow\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\nThis is a cow .\n\n7', metadata={'source': '/tmp/tmpcixox6eb/documents/cow.docx', 'country': 'country0'}), Document(page_content='This is a dog . This is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\nThis is a dog .\n\n7', metadata={'source': '/tmp/tmpv_2j4w3x/documents/dog.docx', 'country': 'country1'}), Document(page_content='This is a horse . This is a horse .\n\nThis is a horse\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\n7', metadata={'source': '/tmp/tmp48s0chjy/documents/dogggg.docx', 'country': 'country2'}), Document(page_content='This is a horse . This is a horse .\n\nThis is a horse\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\nThis is a horse .\n\n7', metadata={'source': '/tmp/tmp_wl23tp6/documents/horse.docx', 'country': 'country3'})]

[npthao1312]

@dosu-bot Hi, how can I filter metadata in opensearch javascript, for example, I want to filter metadata.source with multiple file names?

[yynmsl]

@npthao1312 use boolean_filter to query you data

vector.similarity_search(question, k=k, index_name=index_name,
                                             vector_field="embedding",
                                             text_field="content",
                                             verbose=True,
                                             boolean_filter={"bool": {"should": [
                                                 {"match": {"tenant_id": tenant_id}},
                                             ]}}
                                             )