Is there any way to read from memory instead of from a file?

[jamesleeht]

Currently, SimpleDirectoryReader allows us to read from a local filepath.

However, in some applications you might want to take a bytearray of a file that's already in memory and allow LlamaIndex to read this - the alternative would be to save the data into a file, use SimpleDirectoryReader, then delete the file.

Is there any way to achieve this?

[dosubot[bot]]

Based on the current implementation of SimpleDirectoryReader in LlamaIndex, it primarily focuses on reading files from a directory or a list of file paths. The methods provided, such as load_data and load_file, are designed to work with file paths (input_file as a Path object) and do not directly support reading from a bytearray or an in-memory file.

To achieve reading from a bytearray or an in-memory file, you would need to extend or modify the existing SimpleDirectoryReader class or the specific file reader classes (e.g., PDFReader, DocxReader) to include a method that accepts a bytearray or an in-memory file object instead of a file path. This new method would bypass the need for file system operations and directly process the in-memory data.

Here's a conceptual example of how you might implement a method to read from a bytearray for a hypothetical InMemoryReader class, which you would need to integrate into the LlamaIndex framework:

from io import BytesIO
from llama_index.core.readers.base import BaseReader
from llama_index.core.schema import Document

class InMemoryReader(BaseReader):
    def read_from_bytearray(self, data: bytearray, metadata: dict = None) -> List[Document]:
        # Convert bytearray to a file-like object
        file_like_object = BytesIO(data)
        
        # Process the file-like object as needed
        # This is a placeholder for actual processing logic, which would vary based on file format
        text = file_like_object.read().decode('utf-8')
        
        # Create and return a Document object
        doc = Document(text=text, metadata=metadata or {})
        return [doc]

# Example usage
data = bytearray(b"Example content of the file in memory")
reader = InMemoryReader()
documents = reader.read_from_bytearray(data)

This example is a simplified illustration. The actual implementation would depend on the specific file formats you need to support and how their respective reader classes in LlamaIndex process files. You would need to adapt the file processing logic within read_from_bytearray to handle different formats accordingly.

Please note that this is a conceptual example and might require adjustments based on your specific use case and the file formats you are working with. You can refer to the source code of the SimpleDirectoryReader and its subclasses in the LlamaIndex repository for more details on how they process files.

Sources

• llama-index-core/llama_index/core/readers/file/base.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Cherchercher]

Did you figure it out? I'm trying to read and process from the streamlit file uploader, which gives byteIO...

[Ganesh-Biogen]

yes, how to use llamaindex loaders with streamlit file uploaded? It just returns the bytes, not the filepath ☹️

[mario-holubar]

I worked around this by dumping uploaded files into a temporary folder that immediately gets deleted afterwards:

import os
import uuid

# Create a temporary folder to store in-memory files in, which is removed after use.
# Takes a dictionary of the form {filename: bytes} so it also works for non-streamlit applications.
# Note that this removes all file metadata.
class TempDir:
    def __init__(self, files):
        # This assumes Unix-like filesystem.
        # Consider using the tempfile module to make it cross-platform
        self.tmpdir = os.path.join("/tmp/upload/", str(uuid.uuid4()))
        self.files = files

    def __enter__(self):
        os.makedirs(self.tmpdir)
        for filename, file_bytes in self.files.items():
            file_path = os.path.join(self.tmpdir, filename)
            with open(file_path, 'wb') as f:
                f.write(file_bytes)
        return self.tmpdir

    def __exit__(self, exc_type, exc_value, exc_traceback):
        for filename in self.files.keys():
            file_path = os.path.join(self.tmpdir, filename)
            os.remove(file_path)
        os.rmdir(self.tmpdir)
        return False

Use it like this:

file_dict = {file.name: file.getvalue() for file in uploaded_files}
with TempDir(file_dict) as tempdir:
    reader = SimpleDirectoryReader(input_dir=tempdir)

This is probably the most painless way to go about it, but I would definitely prefer to have a canonical solution to this problem.

[tooluser]

The Document class is exposed.

from llama_index.core import Document

text_list = [text1, text2, ...]
documents = [Document(text=t) for t in text_list]