Skip to content

Commit

Permalink
[Housekeeping] stop support the python3.8 (#2909)
Browse files Browse the repository at this point in the history 
* feat: stop support the python3.8

Signed-off-by: terryhung <[email protected]>

* fix the image hash

Signed-off-by: terryhung <[email protected]>

* update the pyproject.toml

Signed-off-by: terryhung <[email protected]>

* add the  python3.9 support back to flytekit-async-fsspec

Signed-off-by: terryhung <[email protected]>

---------

Signed-off-by: terryhung <[email protected]>
  • Loading branch information
@Terryhung
Terryhung authored Nov 7, 2024
1 parent 51f9a3e commit 3475ddc
Show file tree
Hide file tree
Showing 53 changed files with 61 additions and 114 deletions.
4 changes: 1 addition & 3 deletions .github/workflows/pythonbuild.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
- name: Set Python versions for run
run: |
if [[ ${{ github.event_name }} == "schedule" ]]; then
echo "python_versions=[\"3.8\",\"3.9\",\"3.10\",\"3.11\",\"3.12\"]" >> $GITHUB_ENV
echo "python_versions=[\"3.9\",\"3.10\",\"3.11\",\"3.12\"]" >> $GITHUB_ENV
else
echo "python_versions=[\"3.9\", \"3.12\"]" >> $GITHUB_ENV
fi
Expand Down Expand Up @@ -363,8 +363,6 @@ jobs:
- flytekit-vaex
- flytekit-whylogs
exclude:
- python-version: 3.8
plugin-names: "flytekit-aws-sagemaker"
- python-version: 3.9
plugin-names: "flytekit-aws-sagemaker"
# flytekit-modin depends on ray which does not have a 3.11 wheel yet.
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/pythonpublish.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,6 @@ jobs:
strategy:
matrix:
python-version:
- "3.8"
- "3.9"
- "3.10"
- "3.11"
Expand Down
7 changes: 3 additions & 4 deletions plugins/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ All the Flytekit plugins maintained by the core team are added here. It is not n
| dbt | `bash pip install flytekitplugins-dbt` | Run dbt within Flyte | [![PyPI version fury.io](https://badge.fury.io/py/flytekitplugins-dbt.svg)](https://pypi.python.org/pypi/flytekitplugins-dbt/) | Flytekit-only |
| Huggingface | `bash pip install flytekitplugins-huggingface` | Read & write Hugginface Datasets as Flyte StructuredDatasets | [![PyPI version fury.io](https://badge.fury.io/py/flytekitplugins-huggingface.svg)](https://pypi.python.org/pypi/flytekitplugins-huggingface/) | Flytekit-only |
| DuckDB | `bash pip install flytekitplugins-duckdb` | Run analytical workloads with ease using DuckDB | [![PyPI version fury.io](https://badge.fury.io/py/flytekitplugins-duckdb.svg)](https://pypi.python.org/pypi/flytekitplugins-duckdb/) | Flytekit-only |
| ChatGPT | `bash pip install flytekitplugins-openai` | Interact with OpenAI's ChatGPT. | [![PyPI version fury.io](https://badge.fury.io/py/flytekitplugins-openai.svg)](https://pypi.python.org/pypi/flytekitplugins-openai/) | Flytekit-only |
| OpenAI Batch | `bash pip install flytekitplugins-openai` | Submit requests to OpenAI for asynchronous batch processing. | [![PyPI version fury.io](https://badge.fury.io/py/flytekitplugins-openai.svg)](https://pypi.python.org/pypi/flytekitplugins-openai/) | Flytekit-only |
| ChatGPT | `bash pip install flytekitplugins-openai` | Interact with OpenAI's ChatGPT. | [![PyPI version fury.io](https://badge.fury.io/py/flytekitplugins-openai.svg)](https://pypi.python.org/pypi/flytekitplugins-openai/) | Flytekit-only |
| OpenAI Batch | `bash pip install flytekitplugins-openai` | Submit requests to OpenAI for asynchronous batch processing. | [![PyPI version fury.io](https://badge.fury.io/py/flytekitplugins-openai.svg)](https://pypi.python.org/pypi/flytekitplugins-openai/) | Flytekit-only |

## Have a Plugin Idea? 💡

Expand Down Expand Up @@ -101,12 +101,11 @@ setup(
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-airflow/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-async-fsspec/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}", f"flytekitplugins.{PLUGIN_NAME}.s3fs"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-aws-athena/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-aws-batch/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-bigquery/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-comet-ml/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,11 @@
packages=[f"flytekitplugins.{MODULE_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-dask/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8", # dask requires >= 3.8
python_requires=">=3.9", # dask requires >= 3.9
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-data-fsspec/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,11 @@
"gcp": [],
},
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-dbt/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-deck-standard/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,11 @@
install_requires=plugin_requires,
extras_require=extras,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-dolt/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,11 @@ def run(self):
),
cmdclass=dict(develop=PostDevelopCommand),
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
4 changes: 1 addition & 3 deletions plugins/flytekit-duckdb/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.7,<3.13",
python_requires=">=3.9,<3.13",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-envd/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
12 changes: 6 additions & 6 deletions plugins/flytekit-envd/tests/test_image_spec.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,13 @@ def register_envd_higher_priority():
def test_image_spec():
base_image = ImageSpec(
packages=["numpy"],
python_version="3.8",
python_version="3.9",
registry="",
base_image="cr.flyte.org/flyteorg/flytekit:py3.8-latest",
base_image="cr.flyte.org/flyteorg/flytekit:py3.9-latest",
)
# Replace the base image name with the default flytekit image name,
# so Envd can find the base image when building imageSpec below
ImageBuildEngine._IMAGE_NAME_TO_REAL_NAME[base_image.image_name()] = "cr.flyte.org/flyteorg/flytekit:py3.8-latest"
ImageBuildEngine._IMAGE_NAME_TO_REAL_NAME[base_image.image_name()] = "cr.flyte.org/flyteorg/flytekit:py3.9-latest"

with tempfile.TemporaryDirectory(dir=Path.cwd().as_posix()) as temp_dir:
copy_file = Path(temp_dir) / "copy_file.txt"
Expand All @@ -39,7 +39,7 @@ def test_image_spec():
image_spec = ImageSpec(
packages=["pandas"],
apt_packages=["git"],
python_version="3.8",
python_version="3.9",
base_image=base_image,
pip_index="https://pypi.python.org/simple",
source_root=os.path.dirname(os.path.realpath(__file__)),
Expand All @@ -57,13 +57,13 @@ def test_image_spec():
== f"""# syntax=v1
def build():
base(image="cr.flyte.org/flyteorg/flytekit:py3.8-latest", dev=False)
base(image="cr.flyte.org/flyteorg/flytekit:py3.9-latest", dev=False)
run(commands=["echo hello"])
install.python_packages(name=["pandas"])
install.apt_packages(name=["git"])
runtime.environ(env={{'PYTHONPATH': '/root:', '_F_IMG_ID': '{image_spec.id}'}}, extra_path=['/root'])
config.pip_index(url="https://pypi.python.org/simple")
install.python(version="3.8")
install.python(version="3.9")
io.copy(source="./", target="/root")
io.copy(source="{copy_file.relative_to(Path.cwd()).as_posix()}", target="/root/{copy_file.parent.relative_to(Path.cwd()).as_posix()}/")
"""
Expand Down
4 changes: 2 additions & 2 deletions plugins/flytekit-flyteinteractive/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def train():
```

## User Guide
1. Build the image with Dockerfile.dev `docker buildx build --push . -f Dockerfile.dev -t localhost:30000/flytekit:dev --build-arg PYTHON_VERSION=3.8`
1. Build the image with Dockerfile.dev `docker buildx build --push . -f Dockerfile.dev -t localhost:30000/flytekit:dev --build-arg PYTHON_VERSION=3.9`
2. Run the decorated task on the remote. For example: `pyflyte run --remote --image localhost:30000/flytekit:dev [PYTHONFILE] [WORKFLOW|TASK] [ARGS]...`
3. Once the code server is prepared, you can forward a local port to the pod. For example: `kubectl port-forward -n [NAMESPACE] [PODNAME] 8080:8080`.
4. You can access the server by opening a web browser and navigating to `localhost:8080`.
Expand Down Expand Up @@ -120,7 +120,7 @@ def train():
```

## User Guide
1. Build the image with Dockerfile.dev `docker buildx build --push . -f Dockerfile.dev -t localhost:30000/flytekit:dev --build-arg PYTHON_VERSION=3.8`
1. Build the image with Dockerfile.dev `docker buildx build --push . -f Dockerfile.dev -t localhost:30000/flytekit:dev --build-arg PYTHON_VERSION=3.9`
2. Run the decorated task on the remote. For example: `pyflyte run --remote --image localhost:30000/flytekit:dev [PYTHONFILE] [WORKFLOW|TASK] [ARGS]...`
3. Once the code server is prepared, you can forward a local port to the pod. For example: `kubectl port-forward -n [NAMESPACE] [PODNAME] 8888:8888`.
4. You can access the server by opening a web browser and navigating to `localhost:8888`.
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-flyteinteractive/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,11 @@
],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-greatexpectations/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-hive/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-huggingface/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-inference/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,11 @@
],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
3 changes: 1 addition & 2 deletions plugins/flytekit-k8s-pod/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
4 changes: 1 addition & 3 deletions plugins/flytekit-kf-mpi/setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,11 @@
packages=[f"flytekitplugins.{PLUGIN_NAME}"],
install_requires=plugin_requires,
license="apache2",
python_requires=">=3.8",
python_requires=">=3.9",
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering",
Expand Down
Loading

0 comments on commit 3475ddc

Please sign in to comment.