Skip to content
warpforgedotcoPublic

About

CPython's statically linked, as a patch series (started from MonolithPy)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

33 Commits

Folders and files

Repository files navigation

kpython

A fully statically linked CPython, kept as series of patches on top of CPython's latest releases instead of as fork branches: one series per CPython release series. The patches started as MonolithPy, split by topic.

This repository holds only:

  • tools/vendoring/cpython-<series>/upstream.txt: the CPython release tag a series' patches apply to, such as v3.14.10.
  • tools/vendoring/cpython-<series>/patches/*.patch: that series' patches, applied in name order.
  • scripts/vendor_cpython.py: builds a series' patched tree into build/<series>/cpython, which is never committed.
  • scripts/import_monolith.py: started the 3.14 series from MonolithPy. It isn't needed to maintain the series.

Working on the patches

uv run scripts/vendor_cpython.py sync --series 3.14     # move to the latest 3.14 release
uv run scripts/vendor_cpython.py check                  # every series applies to its pin, byte for byte as exported
uv run scripts/vendor_cpython.py export --series 3.14   # write build/3.14/cpython's commits back as patches
uv run scripts/vendor_cpython.py build --series 3.14    # sync the pin, then run the tree's build script

--series can be left out while there is only one series. Each series has its own checkout and build directory under build/<series>/, so series don't share build state.

build/<series>/cpython is a git checkout with three branches:

  • upstream: the CPython release.
  • base: upstream with Doc/, .github/, .azure-pipelines/ and .devcontainer/ removed. Nothing in the build reads them, and deleting them in a patch would store every deleted line.
  • kpython: base plus one commit per patch.

To change a patch, commit on kpython (git rebase -i base to amend one), then run export and commit the patches.

When a new release comes out, sync fetches it along with the release the patches were exported against, and applies each patch with git am --3way. A patch whose context only moved merges by itself. When one really conflicts, sync stops in build/<series>/cpython:

  1. Resolve the conflict and git add the files.
  2. Run git am --continue.
  3. Apply the remaining patches with git am --3way --keep-cr ../../../tools/vendoring/cpython-<series>/patches/00NN-*.patch.
  4. Run export --series <series>, which also moves the pin.

To start a new series, copy the previous series' directory to tools/vendoring/cpython-<new series> and sync --series <new series>: it merges the copied patches from the old release onto the new one, the same way.

CI runs check on every change. Every day it also tries sync for each series against its latest release, and opens an issue when one fails.

build runs build.mac.sh, build.sh or build.bat from the patched tree. Those are MonolithPy's scripts. They download and statically build the dependencies, and install into ~/Library/MonolithPy<version>-<arch> or /opt/monolithpy<version> unless given a target directory after --.

The patches

Each series has the same patches, numbered the same way. 0001 to 0011 are MonolithPy's changes; 0012 onward are kpython's own, one standalone patch each.

MonolithPy's patches

Patch What it does
0001 PathCch A private PathCch implementation, so Windows 7 can load the interpreter.
0002 Embedded VFS mp_embed: data such as CA certificates linked into the binary and served from a virtual filesystem. Also adds zipfile.ZipFile.read_batch.
0003 Static extensions Every extension module is linked in and registered through staticinit.h, with dynamic loading turned off. The ABI tag becomes mp/monolithpy, sys.implementation.name becomes monolithpy, and thin LTO is used.
0004 Frozen modules Only importlib and io stay frozen, so the working directory can't shadow them at startup.
0005 Relocatable sysconfig Prefixes are rewritten to where the interpreter runs, and the user site is ignored.
0006 Relink on install Lib/__mp__, pyobjtools and rebuildpython.py: installing a package with C extensions builds it statically, renames its symbols and relinks the interpreter.
0007 ensurepip Bundles setuptools, wheel, packaging and pyproject_hooks, and a pip whose vendored pyproject_hooks puts backend-path on sys.path.
0008 Windows build PCbuild builds statically. The dependencies come from Nuitka/MonolithPy-source-deps and -bin-deps.
0009 Unix build build.sh and build.mac.sh.
0010 Tests Adapts the test suite to a single binary that can't start a plain CPython.
0011 Branding MonolithPy's README, icons and NuGet packaging. kpython's patches don't touch its files, so rebranding means replacing this one patch.

They are split by file, not by hunk. A file that serves several topics belongs to the one it serves most, and that patch's message says what else the file carries. Only the whole series is meant to build; a prefix of it may not.

They hold only MonolithPy's changes, carried to each release: in the 3.14 series they are MonolithPy 66fb6eff moved to v3.14.10, and in the 3.15 series the same moved to v3.15.2. Moving to a release resolves conflicts inside them, so that each change does the same thing on the new release. Anything that does something MonolithPy doesn't, such as a bug fix, is a kpython patch instead.

On v3.14.6, 0001 to 0011 reproduce MonolithPy 66fb6eff exactly, except for:

  • Pruned paths: the paths base removes (see above).
  • MonolithPy's own CI: its .github/ workflows are left out.
  • Line endings: Programs/python.c and three Tools/msi/*.wxs files keep upstream's LF. MonolithPy had changed them to CRLF without changing anything else.
  • Find-and-replace renames: these were kept as-is, and are probably side effects of a global cp/np → mp replacement. 0010 renames posix_spawnp_test_program to posix_spawmp_test_program and np_ds_req to mp_ds_req.

kpython's patches

Patch Series What it does
0012 both build.mac.sh merges PGO profiles with Xcode's llvm-profdata when it uses Apple clang. A newer one first on PATH, such as Homebrew's llvm, rejected them with "raw profile version mismatch".
0013 both build.sh and build.mac.sh read the Python version from Include/patchlevel.h. MonolithPy took it from the git branch name (3.14), which breaks on any other branch.
0014 both mp_fdopen returns NULL when fdopen fails, instead of a wrapper around NULL that crashed the next stdio call (numpy's test_fromfile_failed_fdopen_closes_dup segfaulted in ftello), and mp_freopen reopens the caller's stream.
0015 both mp_embed.h declares the Linux stdin, stdout and stderr as glibc's FILE * instead of EFILE *. GCC 14 and later reject the mismatch in dependencies that include it: OpenSSL 3.5.7 failed to build with GCC 15.2.
0016 both Lib/pip.py passes through the keyword arguments pip 26 added (script_executable). Every install failed without it.
0017 both Lib/pip.py hooks pip 26's build_env.installer, so build environments run this pip again. pip 26 made build_env a package, and plain pip can't install mp3_14 wheels such as OpenBLAS.
0018 both Lib/pip.py keys its wheel cache by the wheel's SHA-256 as well as its name. A wheel built from an sdist has the same name as the index's prebuilt one, and installing it put the prebuilt files back.
0019 both Recipe commands put only this build's pip build environments on PATH, and build tools and dependency libraries are looked up only there. MonolithPy used every pip-build-env-* in the temp directory, including other builds', so meson could run a foreign cython that can't import its package: numpy failed with "Unknown compiler(s): cython", or now and then "Compiler cython cannot transpile programs".
0020 both Package recipes come from warpforgedotco/MonolithPy-packages at a fixed commit instead of Nuitka/MonolithPy-packages master. The fork adds Linux recipes and mp315-* ones, and has the macOS numpy and OpenBLAS fixes.
0021 both Recipe builds don't require the clang build tool on Linux, where they use the system compiler and nothing calls clang. Every recipe build there failed for want of a clang tool recipe.
0022 both On Linux the relink links the C++ runtime, statically with -static-libgcc, so packages with C++ code such as numpy link and the interpreter still needs only libc and libm. macOS already linked it.
0023 both On Linux the relink passes the static libraries inside --start-group/--end-group. Its ordering heuristic put libssl before libcrypto once numpy's libraries were added.
0024 both An install whose relink fails is undone: the packages just installed are uninstalled and the versions they replaced restored. Otherwise the package stayed installed without being linked in, and its library broke every later relink.
0025 both MonolithPy's ar reader accepts GNU thin archives (!<thin>), reading each member from the file it names, and reads a long-name offset that GNU ar wrote over a 15-character name (/237 /). On Linux, meson builds the static libraries it doesn't install as thin archives, and scipy failed at libhighs.a with "not a valid ar archive".
0026 both On macOS the PyInit rename gives every global __pyx_/__Pyx_ definition the module's suffix, as MonolithPy did for pyx_CommonTypesMetaclass alone. Cython leaves __pyx_module_is_main_<name> and, in 3.3, __Pyx_PyProbablyModule_GetDict global, and Apple's linker rejected sqlalchemy's duplicates. Linux links with -allow-multiple-definition and is left as it was.
0027 both MonolithPy's pure-Python nm reads LLVM bitcode's symbol table as llvm/Object/IRSymtab.h lays it out. It guessed where the table started and stopped at the first unusual name, and read the visibility bit as "undefined", so on macOS, where packages are LTO bitcode, it saw 4 of a Cython module's symbols. Its global and undefined symbols now match nm's on all 4787 bitcode objects of the installed packages.
0028 both The LTO rename strips Darwin's _ prefix from the linker names it is given before matching IR names (_PyInit_foo is @PyInit_foo). Nothing matched before, so on macOS the PyInit rename and every other rename of an LTO object did nothing.
0029 both A wheel built from a recipe also carries the recipe's dist_requires. pip resolves the first build from the recipe's metadata, but a later install from its wheel cache reads the wheel's, so the requirements were lost: a cached meson-python no longer brought patchelf, and scipy failed to build on Linux.
0030 both On Linux the library rename leaves weak definitions, such as C++ inline functions, as they are. It renamed the members of their COMDAT groups but not the groups' keys, so the linker kept one library's group and dropped another's, whose renamed members went undefined: scipy's relink failed on _highspy._core's HighsOptions destructor.
0031 both On Linux the PyInit rename collects the symbols to rename from the whole archive and renames their references in every object, as on macOS. It renamed each object's definitions only there, so a pybind11 function defined in one object and called from another went undefined: scipy's _fmm_core and init_write_array.
0032 both build.sh builds harfbuzz again with freetype once freetype is installed, as build.mac.sh does. It built it only without, so there was no hb-ft.h and raqm, which the pillow and matplotlib recipes use, failed to compile on Linux.
0033 both build.sh builds expat before fontconfig, which requires it. fontconfig's configure found expat only where the system has its headers (libexpat1-dev), and failed elsewhere.
0034 both On Linux Python.h no longer redefines _GNU_SOURCE, _POSIX_C_SOURCE or _XOPEN_SOURCE. mp_embed.h, which it includes first, defined _GNU_SOURCE without a value and let glibc set the other two before pyconfig.h did, so every extension warned and contourpy, built with werror=true, failed.
0035 both mp_embed.h turns off the compilers' warning that its stdio replacements, which take a void *stream to accept both an EFILE and a FILE, differ from the built-in putc, fputc, fputs and fwrite. With GCC, C++ extensions built in GNU mode with warnings as errors failed.
0036 both On macOS the wheels MonolithPy builds are tagged with the deployment target (macosx_11_0_arm64, from sysconfig.get_platform()) instead of the running system (macosx_26_0). What they hold targets macOS 11.0, like the interpreter, but macOS older than the build machine couldn't install them.
0037 both pip also searches kpython's wheel index, after the configured indexes. For 3.15 MonolithPy's mirror has no wheels; for 3.14 its macOS numpy doesn't link BLAS into matmul, which ran 60 times slower, and the index has kpython's builds. KPYTHON_INDEX_URL overrides it; an empty value turns it off.
0038 both build_package gives every recipe's wheel kpython's platform tag (get_wheel_tag()), whatever backend built it. setuptools named Linux wheels linux_x86_64, which says nothing about the glibc they need.
0039 both build.sh links -lpthread for mp_embed, which uses pthread_once and pthread mutexes. Before 2.34 glibc keeps those in libpthread, so on manylinux_2_28 (glibc 2.28) nothing linked against mp_embed linked.
0040 both build.sh links librt dynamically (-lrt). The interpreter's LIBS, which every relink reuses, named glibc's static librt.a; RHEL keeps that in the optional glibc-static, so on RHEL 8 installing a package that needs a relink failed. The interpreter loads librt.so.1 all the same.
0041 3.15 build.sh and build.mac.sh build the stub dynamic loader. 3.15's dynload_shlib.c no longer compiles with dynamic loading turned off.
0042 3.15 Extension modules that export a PEP 793 hook (PyModExport_<name>) instead of PyInit_<name> are linked in: _PyImport_AppendInittabExport() registers them, and the relink tooling finds and registers them.

Moving to v3.14.8

Decisions made when moving from v3.14.6 to v3.14.8:

  • Tcl/Tk 8.6.15 and OpenSSL 3.5.7 on Windows: upstream moved to Tcl/Tk 9.0.4 and OpenSSL 3.5.9. Nuitka/MonolithPy-*-deps has no static builds of those yet.
    • PCbuild/tcltk.props and PCbuild/python.props had merged to upstream's versions without a conflict, so they were set back by hand.
    • Upstream's other Windows changes were kept: the v145 toolset, the ARM64 compiler workaround and the clang COMPILER string.
  • pip 26.2.1: upstream's new pip, with MonolithPy's one-line pyproject_hooks change applied again and the wheel's RECORD updated to match. Lib/pip.py needed 0016 and 0017 for it.
  • _tkinter.c: upstream's new Tk 9 tkPlatDecls.h include is inside the STATIC_BUILD region.

Moving to v3.15.0

The 3.15 series started as a copy of the 3.14 series, merged onto v3.15.0. Decisions made in MonolithPy's patches:

  • sys.implementation.name (0003): 3.15 lets the build set the name with _PY_IMPL_NAME. Its default is now monolithpy, in place of MonolithPy's hard-coded name; the cache tag (monolithpy-315) follows from it.
  • BuiltinImporter (0003): keeps MonolithPy's iter_modules(). 3.15 removed load_module() and _load_module_shim, so it is gone here too.
  • _decimal.c (0003): MonolithPy's change is dropped. It made each CONFIG_32/CONFIG_64 test also accept MPD_CONFIG_*, which the file's own mapping already does, in 3.14.8 as well.
  • Module lists (0003): Modules/Setup and Setup.macos add 3.15's _math_integer, list _remote_debugging's twelve source files and link it with zstd (-DHAVE_ZSTD), and disable the xxlimited_3_13 example.
  • Frozen modules (0003, 0004): 3.15 also freezes linecache; it stays unfrozen, like the other modules MonolithPy doesn't freeze. The installed test directories keep MonolithPy's removal of test_json and add 3.15's test_io and test_lazy_import.
  • ABI tag (0005): MonolithPy's sysconfig sets SOABI to the literal mp3.14; it is mp3.15 here, which pip's wheel tags follow.
  • Lib/pip.py and Lib/wheel.py (0006): they loaded the real pip and wheel with SourceFileLoader.load_module(), which 3.15 removed, and now load them from a spec. MonolithPy's code ran each package's __init__ twice; it runs once.
  • ensurepip (0007): 3.15 bundles the same pip 26.2.1, so the wheels are the same as in the 3.14 series. MonolithPy's ArgumentParser(prog="python -m ensurepip") was 3.13's line; 3.15's is kept.
  • Windows (0008), untested:
    • Dependencies stay at what Nuitka/MonolithPy-*-deps provides, as in 3.14. sqlite 3.53.4.0 and xz 5.8.1.1 had merged into python.props and Misc/externals.spdx.json without a conflict and were set back to 3.50.4.0 and 5.2.5.
    • _remote_debugging now needs ntdll.lib, which python.vcxproj links, as extensions are static libraries.
    • 3.15's python3t.dll (the free-threaded stable ABI DLL) and its installer components are left out, like python3.dll.
    • build.bat forces Release for win32 on 3.15's new PGO path as well.
    • pyexpat follows 3.15 in dropping PYEXPAT_EXPORTS.
    • LLVM 21 is only fetched for JIT builds, which MonolithPy doesn't make.
    • PC/layout/main.py names python315.lib where MonolithPy named python314.lib.
  • Tests (0010): test_posix.py moved to Lib/test/test_os/.

Moving to v3.14.10

Decisions made when moving from v3.14.8 to v3.14.10:

  • Module lists (0003): Modules/Setup keeps MonolithPy's static _zstd line; upstream's change to the commented-out one, which now lists all four _zstd sources, was already in it. Modules/Setup.stdlib.in stays MonolithPy's empty stub; upstream's new _testlimitedcapi sources (type.c, build.c) belong to a test module the static build disables.
  • pip 26.2.2 (0007): upstream's new pip, with MonolithPy's one-line pyproject_hooks change applied again and the wheel's RECORD updated to match. Every other entry of the wheel is upstream's, with its timestamps and compression; the same rewrite reproduces 3.14.8's modified pip 26.2.1 byte for byte. _PIP_VERSION in MonolithPy's ensurepip/__init__.py is 26.2.2.
  • Windows (0008), untested:
    • PCbuild/pyproject-clangcl.props keeps MonolithPy's removal of -flto=thin, and takes upstream's new -Xclang -Og for Debug builds.
    • Upstream's new Modules/expat/xcs.c merged into pyexpat.vcxproj and _elementtree.vcxproj, which both still compile the bundled expat, as before.
    • Upstream changed no dependency versions: PCbuild/*.props, get_externals.bat and Misc/externals.spdx.json are as in v3.14.8, so nothing had to be set back.
  • .gitignore (0009): upstream's new /do-not-commit/ entry comes before MonolithPy's block.
  • README.rst (0011): stays MonolithPy's; upstream changed only its version line.

Moving to v3.15.2

Decisions made when moving from v3.15.0 to v3.15.2. Upstream's changes to the files MonolithPy's patches touch were the same as in v3.14.10, and so were the resolutions, apart from ctypes:

  • ctypes.pythonapi (0003): stays None, as MonolithPy has it: a static interpreter has no libpython to load. Upstream now catches OSError from PyDLL(None) and sets None itself; MonolithPy's commented-out copy of upstream's code is updated to match it.
  • Module lists (0003): as in v3.14.10. Modules/Setup already listed all four _zstd sources, and Modules/Setup.stdlib.in stays the empty stub; _testlimitedcapi's new sources are for a disabled test module.
  • pip 26.2.2 (0007): v3.15.2 bundles the same pip wheel as v3.14.10, so the modified wheel is identical to the 3.14 series' one.
  • Windows (0008), untested: pyproject-clangcl.props is resolved as in v3.14.10. Upstream changed no dependency versions, so nothing had to be set back.
  • .gitignore (0009) and README.rst (0011): as in v3.14.10.
  • Merged without a conflict: upstream's import changes in Python/import.c (a submodule's package context for an explicit init function, and lazy-import fixes) and Lib/importlib/_bootstrap.py (parent packages are resolved before a submodule's lock is taken) sit next to MonolithPy's and kpython's changes there (0003, 0042) and needed nothing.

Installing packages

The bundled pip (Lib/pip.py, 0006) uses MonolithPy's services:

  • Its default index is MonolithPy's mirror, https://pip-mpy.ohrtech.net/main/. It serves prebuilt static wheels for 3.14, for example numpy 2.5.3 with OpenBLAS, from a Backblaze bucket. It has no 3.15 wheels.
  • Per-package build recipes come from warpforgedotco/MonolithPy-packages at a fixed commit (0020), a fork of Nuitka/MonolithPy-packages with recipes for Linux and for 3.15. The build tools' recipes download prebuilt releases, so on 3.15, which the mirror has no tool wheels for, they don't compile anything.
  • pip also searches kpython's own index, https://warpforgedotco.github.io/kpython-wheels/simple/ (0037), which holds wheels built here with those recipes. Its wheels are release assets of warpforgedotco/kpython-wheels, published with that repository's scripts/publish.py.
  • The fork's recipes cover numpy, scipy, pandas, scikit-learn, statsmodels, matplotlib, contourpy, pillow, kiwisolver, cffi, certifi and jedi on macOS and Linux, and pyyaml (with libyaml), multidict, frozenlist, propcache, yarl and aiohttp. On Mac 3.15, Linux 3.14 and Linux 3.15 the first twelve install from source together and pass their recipes' tests, and the interpreter still links only system libraries (libc and libm on Linux). The others pass on Mac 3.14; sqlalchemy needs no recipe.
  • On Linux numpy is built without SVML, its AVX-512 math in assembly, which the static build doesn't carry, and scikit-learn is built without OpenMP everywhere, as there is no static OpenMP runtime to link: its Cython loops run on one thread, while BLAS keeps its own. Packages without a recipe that build with meson-python fail, as stock meson-python rejects the .a extension suffix.
  • The aiohttp family build their C extensions only on CPython, by sys.implementation.name; their recipes accept monolithpy too, as installing them without a recipe gives pure Python.
  • The index's Linux wheels were built on Ubuntu 26.04 and are tagged manylinux_2_43, so pip installs them only where glibc is 2.43 or newer.
  • On Linux the relink runs LTO as one partition and takes several minutes once numpy is in.
  • If a relink fails, the install is undone (0024): the new packages are uninstalled, the versions they replaced come back, and the interpreter is unchanged.

A package with a C extension and no recipe is built from source, and its extension is linked in (tested with markupsafe). A Rust extension without a recipe isn't supported. hypothesis 6.170 builds a dylib, and the relink that includes it fails its startup check, so the interpreter is kept as it was but the package is left without its native module.

The mirror serves only wheels. To build a package from source, take its sdist from PyPI and keep the mirror for the recipe's tools and dependencies:

python3.14 -m pip install --no-binary numpy \
    --index-url https://pypi.org/simple/ \
    --extra-index-url https://pip-mpy.ohrtech.net/main/simple/ numpy==2.5.3

The recipe patches the package for static linking, and its patch is written for particular versions: numpy 2.5.3 and 2.5.4 have patches of their own, as 2.5.4's vendored meson rejects the 2.5.3 one. The toolchain (clang 19, gcc 14, cmake) and libraries such as OpenBLAS still come from the mirror as prebuilt wheels, though meson used Xcode's clang for numpy.

numpy 2.5.3, prebuilt or built here, passes its own test suite apart from tests that compile extensions with meson, that read a compiled module's __file__, or that run numpy-config from PATH.

Licenses

CPython is under the PSF License. Lib/__mp__/tools/pyobjtools is under MPL-2.0 (its LICENSE is in patch 0006). The patches keep their MonolithPy authors.

About

CPython's statically linked, as a patch series (started from MonolithPy)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages