How to connect AWS S3-> Lambda handler -> Zilliz

[jahidhasanlinix]

Hi, I have one question, I have data in the S3 bucket, now I want to trigger S3 data using Lambda handler and then initiate my embedding services (chunking and embed) and then store in zilliz. Do you have any better way to set this up, any documentation or code examples? Thank you.

[yhmo]

Vector Transmission Services (VTS) Milvus provides a set of tools for you to transfer your data between a Milvus instance and a bunch of data sources. I suppose this tool can meet your requirements.
https://github.com/zilliztech/vts

Wait the feature owner @nianliuu to comment.

[jahidhasanlinix]

Thank you, I was looking at the code, it's empty https://github.com/zilliztech/vts/blob/master/docs/zilliz/Milvus.md no information to understand how to use it:)

[lentitude2tk]

If your data is ready to be imported into Zilliz, you can use the import interface provided by Zilliz Cloud to complete this step. Refer to the following documentation:

If using RESTful API:
https://docs.zilliz.com/docs/import-data-via-restful-api
https://docs.zilliz.com/reference/restful/create-import-jobs-v2

If using SDKs (e.g., milvus-sdk-java or pymilvus):
https://docs.zilliz.com/docs/import-data-via-sdks

Choose the method that best fits your needs to proceed with the data import process. Hope this is helpful to you! @jahidhasanlinix

[jahidhasanlinix]

Yes, I'm using the cloud zilliz one and it provided the URI and token to use. I'm not using SDK or restful api.
My question is, my lambda is not able to initiate with embedding service and store to zilliz now. Any code examples shows the setup to work on, as my S3 have all data now. But can't store to zilliz.

[nianliuu]

VTS will support migrate from raw data like PDF, image next year. For now, it can only migrate vector data. See bulk writer and import, or https://zilliz.com/zilliz-cloud-pipelines which can help you do the chunking and embedding, etc.

[jahidhasanlinix]

So that means it doesn't work with pdf etc, any other way to do it? As I'm planning to use zilliz not other vectorDB. Also does it does real time data streaming millions of data flow through lambda to zilliz? Any code examples to understand how to set it up from S3 to lambda to embedding and zilliz collection store?

[jahidhasanlinix]

Thank you for all of your answer. But, I have still a bit of doubts, so what I'm trying to do is I have data in S3 that gets real time updated currently. And I have lambda function handler to trigger it and initiate my embedding service which I created an api service to initiate with using LLMAINDEX pipeline to do the chunking and embedding, and then I have zilliz cloud uri and collection name, problem is I can't store it there now. I want once lambda initiate it should call my embedding service and Zilliz service to store vector index! Any help, what is best way I can do this steps to store my data into vector DB. Or Zilliz is not capable of doing real time data streaming? Any suggestions.

[nianliuu]

Can not you just call insert in your llamaIndex ingestion pipeline?
it's supporting to store vector data into zilliz directly I think. https://docs.llamaindex.ai/en/stable/module_guides/loading/ingestion_pipeline/.
Here is the doc for how to connect llamaIndex and Zilliz: https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores/.

You can also call milvus restful api once you called embedding api and got the vector data: https://docs.zilliz.com/reference/restful/insert-v2

[jahidhasanlinix]

Thank you, good question. I actually created separate service for each component to make the codebase more scalable. So, I created one for S3 bucket service, and trigger it in my lambda function, then I initiate my embedding service using the api in lambda to read the metadata, and I already created api server for embedding, I am using LLMAIndex orchestration for embedding process, and I also separately created zilliz service using URI, token and collection name, so once embedding done it insert it to zilliz, but it failed:/ I was trying to look the codebase to find a sample code example for how aws S3 bucket> lambda> embedding+zilliz store work, but couldn't find any. I might need to look more, but nothing particular that I found so far:)

[jahidhasanlinix]

Because I'm using webhook, my data is updating into S3 bucket in real time (till this all working perfectly, my server api works), and I'm trying to make data streaming ingestion though lambda to initiate real time streaming embedding process to do and store to zilliz

[nianliuu]

"so once embedding done it insert it to zilliz, but it failed". Where did you trigger this? In AWS lambda? Is there any error log? @codingjaguar Ping our Head of AI for any suggestions?