Install git submodules

[ReHoss]

My pyproject.toml file contains

requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build]
packages = ["control_dde", "control_dde/mpc/iCEM/icem"]

[tool.hatch.envs.default]
pre-install-commands = ["git submodule update --init --recursive"]

it looks like when I run pip install . the pre-install command is ignored. Why? It works with hatch environments.

More generally, how to do packaging/installation where the customer does not have to run git submodule update --init --recursive before installing with pip?

I would like something which fetches the submodule source code and then add it to the package...

Thank you,
Best,

[ofek]

It works with environments because it's a feature of Hatch only. How are you distributing your code to customers?

Also, the packages option should only be used specifically for the wheel target as the example shows here: https://hatch.pypa.io/latest/config/build/#packages

[ReHoss]

Hi, thanks for your quick answer,

It works with environments because it's a feature of Hatch only. How are you distributing your code to customers?

The project is uploaded on a Github repository (it is a machine learning research project). The customer should clone the repository, activate his virtual environment and then pip install . (this is a very standard approach). I want to hide the git submodule update --init --recursive from the customer interface. What are the solutions?

Also, the packages option should only be used specifically for the wheel target as the example shows here: https://hatch.pypa.io/latest/config/build/#packages

Indeed, I forgot to set tool.hatch.build.target.wheel. However, I think it is unfortunately not stressed enough in the documentation e.g. why sdist for only-include and wheel for packages? Initially, I have been using the include keyword, does it fit more with my use case?:)

Thank you very much,

Note that I have a basic knowledge of packaging.

[ofek]

That repository link doesn't work for me.

For projects as you describe the most common approach would be to enumerate the steps required in a README file or require a tool such as Hatch to reduce the number of steps required (but not eliminate). In the latter case, the interface would be through Hatch and you wouldn't really even want a packaging step so you can remove the wheel target file inclusion options and instead use the dev-mode-dirs option like so:

[tool.hatch.build.targets.wheel]
dev-mode-dirs = ["src"]  # or `.` if your code is at the root

[ReHoss]

That repository link doesn't work for me.

For projects as you describe the most common approach would be to enumerate the steps required in a README file or require a tool such as Hatch to reduce the number of steps required (but not eliminate). In the latter case, the interface would be through Hatch and you wouldn't really even want a packaging step so you can remove the wheel target file inclusion options and instead use the dev-mode-dirs option like so:

[tool.hatch.build.targets.wheel]
dev-mode-dirs = ["src"]  # or `.` if your code is at the root

Thank you.
Could you explain the difference between using pip install . with the hatchling back-end specified in pyproject.toml (which does not execute the pre-install script) vs. using hatch (which works well for pre-install scripts)?

[ofek]

When you do pip install . that builds a wheel using the chosen backend and then essentially does pip install path/to/wheel (same goes for pip install -e . which builds a special wheel).

With Hatch, the default behavior for environments is to basically run that command but with extra features. One such feature is the ability to run some commands before or after that step. If you just have some code you want to run rather than shipping something to users it's better to just define dependencies and put some directories on PYTHONPATH using the option I showed you. You can still structure your code like a project that would be distributed if you want to in which case the directory to where your code is will be derived automatically in the case of editable installs.

[ReHoss]

Thank you very much @ofek . It starts to be clearer to me.

Going back to my initial question:

• Suppose I want to use the pip frontend.
• I want to use the hatchling backend specified by a pyproject.toml (quite common usecase).
• Suppose I have a repository source tree as below:

├── mypkg
│   └── __init__.py
├── pyproject.toml
└── submodules
    └── submodule1

• Suppose the client runs git clone <repository> (note here submodules should not be initialised on the client side)

Is there a way to write a pyproject.toml file such that git submodule update --init is run somehow during pip install .?
If not, does creating a setup.py offers a solution? (I don't know if pip will read both the toml and the setup.py).

Otherwise, are the below client instructions the only solution when working with submodules?

`git submodule update --init`
`pip install .`

Note that replacing pip with hatch with a pre-install command instruction solved this question.

Thank you very much,
Best,

[ReHoss]

That repository link doesn't work for me.

For projects as you describe the most common approach would be to enumerate the steps required in a README file or require a tool such as Hatch to reduce the number of steps required (but not eliminate). In the latter case, the interface would be through Hatch and you wouldn't really even want a packaging step so you can remove the wheel target file inclusion options and instead use the dev-mode-dirs option like so:

[tool.hatch.build.targets.wheel]
dev-mode-dirs = ["src"]  # or `.` if your code is at the root

Here you have kept targets.wheel option, is it a typo?