Add pip req arm64 - #5468
Add pip req arm64#5468JuanMBriones wants to merge 1 commit into
Conversation
When packaging bot deployment bundles on a Linux host via `butler package`, `_install_pip()` installs all packages from `Pipfile.lock` (including transitive dependencies with Linux x86_64 ELF shared objects) into `src/third_party`. `_install_platform_pip()` then downloads pre-compiled binary wheels for each package listed in `src/platform_requirements.txt` to replace the Linux binaries with target platform native binaries. Previously, `markupsafe` and `pynacl` were transitive dependencies containing compiled C/CFFI extensions that were missing from `src/platform_requirements.txt`: * `markupsafe==3.0.3` (transitive dependency of `Jinja2==3.1.4` / `Flask`): contains the compiled C extension `markupsafe/_speedups.cpython-*.so`. Without it in `platform_requirements.txt`, the Linux ELF `.so` remained in the macOS ARM64 bundle. * `pynacl==1.6.0` (transitive dependency of `pygithub==1.55`): contains the compiled libsodium CFFI native binary `nacl/_sodium.abi3.so`. Without it in `platform_requirements.txt`, importing `github` / `nacl` on macOS ARM64 fails with `ImportError: dlopen(.../nacl/_sodium.abi3.so): not a mach-o file`. Adding both packages ensures `butler package -p macos_arm64` downloads the native macOS ARM64/universal2 wheels (`markupsafe-3.0.3-cp311-cp311-macosx_11_0_arm64.whl` and `pynacl-1.6.0-cp38-abi3-macosx_10_10_universal2.whl`) into `src/third_party`. Bug: 555853673 Signed-off-by: Manuel Briones <manuelbriones@google.com>
77e44bd to
dd68bf6
Compare
| @@ -1 +1 @@ | |||
| # This file contains dependencies from src/Pipfile that contain native/compiled | |||
There was a problem hiding this comment.
Can we mention here that it also includes transitive dependencies? We should change Pipfile to mention that transitive dependencies should added here as well.
There was a problem hiding this comment.
I left inline comments. But I think they mention it (but not at the files beginning)
| # into src/third_party. | ||
| # | ||
| # If you add or bump a package in Pipfile that has native extensions, make sure | ||
| # to update it here as well. |
There was a problem hiding this comment.
@g-ortuno I think this might be achieved by above's lines? Also, on Pipfile we've got these lines https://github.com/google/clusterfuzz/blob/dd68bf6511544caa1e457b1c8984286e637d56a1/Pipfile#L7-8
There was a problem hiding this comment.
Right, but I don't think it covers the packages this PR is adding. From the PR description:
markupsafe and pynacl were transitive dependencies containing compiled C/CFFI extensions that were missing from src/platform_requirements.txt:
Specifically the part about transitive dependencies.
2 participants
When packaging bot deployment bundles on a Linux host via
butler package,_install_pip()installs all packages fromPipfile.lock(includingtransitive dependencies with Linux x86_64 ELF shared objects) into
src/third_party._install_platform_pip()then downloads pre-compiledbinary wheels for each package listed in
src/platform_requirements.txtto replace the Linux binaries with target platform native binaries.
Previously,
markupsafeandpynaclwere transitive dependencies containingcompiled C/CFFI extensions that were missing from
src/platform_requirements.txt:markupsafe==3.0.3(transitive dependency ofJinja2==3.1.4/Flask):contains the compiled C extension
markupsafe/_speedups.cpython-*.so.Without it in
platform_requirements.txt, the Linux ELF.soremained inthe macOS ARM64 bundle.
pynacl==1.6.0(transitive dependency ofpygithub==1.55): contains thecompiled libsodium CFFI native binary
nacl/_sodium.abi3.so. Without it inplatform_requirements.txt, importinggithub/naclon macOS ARM64 failswith
ImportError: dlopen(.../nacl/_sodium.abi3.so): not a mach-o file.Adding both packages ensures
butler package -p macos_arm64downloads thenative macOS ARM64/universal2 wheels (
markupsafe-3.0.3-cp311-cp311-macosx_11_0_arm64.whland
pynacl-1.6.0-cp38-abi3-macosx_10_10_universal2.whl) intosrc/third_party.Bug: 555853673