I've been dealing with python vs debian for the last three hours and am deeply angry with the ecosystem. Solved it is not.
Debian decided you should use venv for everything. But when packages are installed in a venv, random cmake nonsense does not find them. There are apt-get level packages, some things find those, others do not. Names are not consistent. There's a thing called pipx which my console recommended for much the same experience. Also the vestiges of 2 vs 3 are still kicking around in the forms of refusing to find a package based on the number being present or absent.
Whatever c++headerparser might be, I'm left very sure that hacking python out of the build tree and leaving it on the trash heap of history is the proper thing to do.
These tools together solve a fraction of the problen. The other parts of the problem are interfacing with classic c, c++ libraries and handling different hardware and different OSes. It is not even funny how tricky it is to use the same GPU/CUDA versions but with different CPU architectures and hopefully most people dont need to be exposed to it. Sometimes parts of the stack depends on a different version of a c++ library than other parts of the stack. Or some require different kernel modules or CUDA driver settings. But I would be happy if there was a standardized way to at least link to the same C++ libraries, hopefully with the same ABI, across different clusters or different OS versions. Python is so far from solved…
We do something similar with a mix of C++, Fortran, and CUDA dependencies. Using https://cibuildwheel.pypa.io/en/stable/ and the repair functionality to restrict the manylinux to something sensible seems to have handled the complexity - while we mostly use Debian and Ubuntu the Arch folks seemed OK too.
We want to share the wheels more smoothly though and having been looking at pyx for that. Seems promising.
Pixi inherited some ot the bad designs from conda, and conda clearly hadnt been close to solving python packaging either and was digging itself into a black hole over time.
uv is venv + insanely fast pip. I’ve used it every day for 5+ months and I still stare in amazement every time I use it. It’s probably the most joy I’ve ever gotten out of technology.
Debian decided you should use venv for everything. But when packages are installed in a venv, random cmake nonsense does not find them. There are apt-get level packages, some things find those, others do not. Names are not consistent. There's a thing called pipx which my console recommended for much the same experience. Also the vestiges of 2 vs 3 are still kicking around in the forms of refusing to find a package based on the number being present or absent.
Whatever c++headerparser might be, I'm left very sure that hacking python out of the build tree and leaving it on the trash heap of history is the proper thing to do.