Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move first provider (airbyte) to a separate project #45259

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Dec 28, 2024

This is the first step to move providers to separate projects inside our monorepo.

Airbyte is a first provider that is separated out to a project under "providers/<PROVIDER_ID>" directory. This has the nice property that all files belonging to the same provider are under a single directory that is part of the Airflow workspace. For now the code is more complex because we are handling providers being in either "old" or "new" structure, but once we move old providers to the new structure, a lot of code could be removed and simplified.

The new structure for provider code is:

providers
        |- PROVIDER_ID
        |            |- src
        |            |    |-airflow
        |            |            |- providers
        |            |                       |- PROVIDER_ID
        |            |- tests
        |            |      |- providers
        |            |                 |- PROVIDER_ID
        |            |- docs
        |            |     |- .latest-doc-only-changes.txt
        |            |- pyproject.toml
        |            |- CHANGELOG.rst
        |            |- provider.yaml
        |            |- README.rst
        |- PROVIDER_ID2
        ...


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:dev-tools area:production-image Production image improvements and fixes kind:documentation labels Dec 28, 2024
@potiuk potiuk force-pushed the move-first-provider-to-separate-project branch 2 times, most recently from a4187bb to b24b864 Compare December 28, 2024 02:05
This is the first step to move providers to separate
projects inside our monorepo.

Airbyte is a first provider that is separated out to a
project under "providers/<PROVIDER_ID>" directory. This has
the nice property that all files belonging to the same provider
are under a single directory that is part of the Airflow
workspace. For now the code is more complex because we are
handling providers being in either "old" or "new" structure, but
once we move old providers to the new structure, a lot of code
could be removed and simplified.

The new structure for provider code is:

```
providers
        |- PROVIDER_ID
        |            |- src
        |            |    |-airflow
        |            |            |- providers
        |            |                       |- PROVIDER_ID
        |            |- tests
        |            |      |- providers
        |            |                 |- PROVIDER_ID
        |            |- docs
        |            |     |- .latest-doc-only-changes.txt
        |            |- pyproject.toml
        |            |- CHANGELOG.rst
        |            |- provider.yaml
        |            |- README.rst
        |- PROVIDER_ID2
        ...

```
@potiuk potiuk force-pushed the move-first-provider-to-separate-project branch from b24b864 to 1cf19b8 Compare December 28, 2024 02:11
@potiuk
Copy link
Member Author

potiuk commented Dec 28, 2024

This is not yet complete, and I also extract some things out of that to separate PR to make it smaller/simpler. Just wanted to test the image build process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant