Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial draft of Daft dataframe support for DeltaCAT catalog #266

Merged
merged 4 commits into from
Feb 26, 2024
Merged

Conversation

raghumdani
Copy link
Collaborator

@raghumdani raghumdani commented Feb 22, 2024

Basic Example

from deltacat.catalog import default_catalog_impl as dc

dc.initialize(ds=ds)
df = dc.read_table(
    table=TABLE_NAME,
    namespace=self.TABLE_NAMESPACE,
    catalog=self.LOCAL_CATALOG_NAME,
    distributed_dataset_type=DistributedDatasetType.DAFT,
    merge_on_read=False,
    deltacat_storage_kwargs=self.kwargs,
)

df.show()

Part of #265

@raghumdani raghumdani changed the title Initial draft of Daft dataframe support for Andes catalog Initial draft of Daft dataframe support for DeltaCAT catalog Feb 22, 2024
deltacat/storage/model/types.py Outdated Show resolved Hide resolved
deltacat/compute/merge_on_read/utils/delta.py Outdated Show resolved Hide resolved
deltacat/storage/interface.py Show resolved Hide resolved
deltacat/aws/s3u.py Outdated Show resolved Hide resolved
deltacat/aws/s3u.py Outdated Show resolved Hide resolved
deltacat/aws/s3u.py Show resolved Hide resolved
Copy link
Member

@Zyiqin-Miranda Zyiqin-Miranda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@raghumdani raghumdani merged commit 3a259a3 into main Feb 26, 2024
5 checks passed
@raghumdani raghumdani deleted the cleanb branch February 26, 2024 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants