You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9ff87c5
Browse filesBrowse the repository at this point in the historyBrowse files
Split the install-time fix-up out of gather into fixup.py (#48)
`gather --patch` and `--origin` replaced the placeholder prefix at build time, standing in for what an installer would do. This extracts the "install step" into a standalone fixup script that can be included in the distro.
`gather` now leaves the placeholder in, so the gathered tree isn't tied to a location. `fixup.py` replaces it once the tree is where it will live. It runs on any Python 3.12+ other than the one in the tree, and fails if no text file in the tree has the placeholder
One deliberate difference from `--patch`: `fixup.py` only rewrites text files. Binaries are left alone, so their signatures stay valid, and it lists the ones that still contain the placeholder (today the interpreter and the `.pyc` files)
`test` gathers, moves the tree, runs `fixup.py` and then checks it. `--no-fixup` skips that step, which is what a build that needs no install-time step would have to pass. CI runs `test`, `test --no-fixup` and, on Linux, `test --relative-rpath`. The Build artifacts workflow uploads one tarball per platform with the tree and `fixup.py`, instead of trees patched for the runner's paths.
On a moved tree, `fixup.py` turns `runtime.paths` and `metadata.build-details` from FAIL to PASS on both platforms. `metadata.prefix-leaks` still fails (the interpreter has the placeholder compiled in), and so does `developer.python-config` on macOS (a shebang can't hold a path with spaces).
@@ -45,16 +45,21 @@ Some stdlib extension modules need third-party libraries that macOS doesn't prov
45
45
46
46
`deps` downloads each source archive from [cpython-source-deps](https://github.com/python/cpython-source-deps), checks it, and installs only static libraries into deps/install, so the extension modules don't link against anything outside the installation. OpenSSL still has compiled-in paths for its configuration and modules under deps/install. Use `--deps-dir DIR` to put them somewhere else.
47
47
48
-
The key thing to note is which patching approach to specify with `relocatable.py gather`. There are:
48
+
`configure` uses a placeholder `--prefix`. `gather` runs CPython's top-level `make install ENSUREPIP=no` into a staging `DESTDIR` and moves the installed prefix into place, so the gathered tree contains everything a normal install provides (such as `build-details.json` and the unversioned `python3` links). The tree still has the placeholder in it, so it isn't tied to a location and can be packed up as is.
49
49
50
-
-`--patch` which rewrites all found paths of the placeholder `--prefix` argument to the target install directory (which defaults to dist/); this is meant to follow the approach used by conda.
51
-
-`--origin` which uses `patchelf` to do a more targeted path rewrite upfront and then to patch any remaining paths; this is meant to follow the approach used by python-build-standalone.
50
+
Replacing the placeholder is a separate step, done at install time. Once the tree is where it will live, run `fixup.py` on it with any Python 3.12+ other than the one in the tree:
52
51
53
-
Either way, `gather` runs CPython's top-level `make install ENSUREPIP=no` into a staging `DESTDIR` and moves the installed prefix into place, so the gathered tree contains everything a normal install provides (such as `build-details.json` and the unversioned `python3` links). Only the placeholder prefix is rewritten, and patched Mach-O files are re-signed ad hoc, since macOS refuses to run binaries whose code signature no longer matches.
52
+
```shell
53
+
python3 ../prebuilt-cpython/fixup.py dist/
54
+
```
55
+
56
+
`fixup.py` replaces the placeholder with the tree's location in text files only and lists the binary files that still contain it. It never modifies a binary, so signatures stay valid. It fails if no text file in the tree has the placeholder, which is the case for a tree it has already been run on.
57
+
58
+
`gather --relative-rpath` also uses `patchelf` to give the installed ELF files an `$ORIGIN`-relative RPATH, so they load libraries from inside the tree wherever it ends up; this follows what python-build-standalone does at build time. It's Linux only.
54
59
55
60
### Testing a relocated build
56
61
57
-
`relocatable.py test` checks whether a build still works after it has been moved. It gathers the build into a temporary directory (dist/ is left alone), renames that directory to a path of a different length with spaces in it, and runs these checks without patching anything again:
62
+
`relocatable.py test` checks whether a build still works after it has been moved. It gathers the build into a temporary directory (dist/ is left alone), renames that directory to a path of a different length with spaces in it, runs `fixup.py` on it, and runs these checks:
58
63
59
64
- The interpreter starts
60
65
-`sys` and `sysconfig` paths point inside the moved tree
@@ -74,11 +79,18 @@ Each check prints `PASS`, `FAIL`, or `SKIP` (in color on a terminal, unless `NO_
74
79
../prebuilt-cpython/relocatable.py test
75
80
```
76
81
77
-
Use `--strategy origin` to gather with `--origin` (the default is `--patch`), `--full` to also run `python -m test --fast-ci`, `--work-dir DIR` to keep the temporary files (including `gather.log`, the `make install` output), and `--format json` for JSON output.
82
+
Use `--relative-rpath` to gather with `--relative-rpath`, `--no-fixup` to check the moved tree without running `fixup.py` (what a build that needs no install-time step would have to pass), `--full` to also run `python -m test --fast-ci`, `--work-dir DIR` to keep the temporary files (including `gather.log`, the `make install` output), and `--format json` for JSON output.
78
83
79
-
The Relocatability GitHub Actions workflow runs `test` with both strategies against CPython `main` on Linux and macOS, daily and for each pull request. Each run's summary lists the results, and the JSON is attached to the run.
84
+
The Relocatability GitHub Actions workflow runs `test` in each of those modes against CPython `main` on Linux and macOS, daily and for each pull request. Each run's summary lists the results, and the JSON is attached to the run.
85
+
86
+
The Build artifacts workflow is run by hand and also attaches the build to the run: a tarball per platform with the gathered tree in `python/` and `fixup.py` next to it. To try one, unpack it, move `python/` to where it should live, and run `fixup.py` on it:
If the build wasn't configured by `relocatable.py configure`, its `--prefix` is left alone and treated as the old location. If that directory still exists, the `distribution.isolation` check fails, because the moved build might be using files from it without anyone noticing.
93
+
If the build wasn't configured by `relocatable.py configure`, `fixup.py` isn't run, and its `--prefix` is treated as the old location. If that directory still exists, the `distribution.isolation` check fails, because the moved build might be using files from it without anyone noticing.
82
94
83
95
## Project Representatives
84
96
These are people representing projects that have experience with prebuilt binaries for CPython.
0 commit comments