We do something similar with a mix of C++, Fortran, and CUDA dependencies. Using https://cibuildwheel.pypa.io/en/stable/ and the repair functionality to restrict the manylinux to something sensible seems to have handled the complexity - while we mostly use Debian and Ubuntu the Arch folks seemed OK too.

We want to share the wheels more smoothly though and having been looking at pyx for that. Seems promising.