I've recently discovered that LibCST versions 1.8.0 and 1.8.1 introduced a new dependency (on `typing_extensions`) without declaring it, so these releases were completely broken in virtualenvs that didn't happen to have this package. How this happened is a story for another day.
To fix this, I decided to bite the bullet on figuring out how to test the wheels built by `cibuildwheel`, which was simple enough - in `pyproject.toml` all I had to add was:
```toml
[tool.cibuildwheel]
test-command = [
"python --version",
"python -m libcst.tool list",
"python -m libcst.tool codemod remove_unused_imports.RemoveUnusedImportsCommand {project}/libcst/_nodes",
]
```
Which is just a list of commands that gets executed in an environment with only the built wheel installed.
Unfortunately this new testing highlighted an additional problem, but only for CPython 3.14 (prerelease) wheels on 32bit linux (that's i686 and armv7l for LibCST):
```
sh: line 1: 1314 Segmentation fault (core dumped) python -m libcst.tool codemod remove_unused_imports.RemoveUnusedImportsCommand /project/libcst/_nodes
cibuildwheel: error: Command ['sh', '-c', 'python --version && python -m libcst.tool list && python -m libcst.tool codemod remove_unused_imports.RemoveUnusedImportsCommand /project/libcst/_nodes'] failed with code 139.
```
## Repro environment
[cibuildwheel](https://cibuildwheel.pypa.io) is awesome enough to let me reproduce this error on an x86_64 Windows machine, using Docker. All I had to do was:
```powershell
$env:CIBW_ENABLE="cpython-prerelease" # enable 3.14 wheels
$env:CIBW_DEBUG_KEEP_CONTAINER="TRUE" # so I can inspect the container
uvx cibuildwheel>=3.0 -only cp314-manylinux_i686
```
Then I could look at the container using `docker ps -al` to find the most recent container ID, and then:
```powershell
docker start $CONTAINER_ID
docker exec -ti $CONTAINER_ID /bin/bash
```
To drop into a shell in the container. To manually repro, I needed to find the virtual environment the test was being run in, but thankfully the logs from that `uvx` command showed a temporary directory being created for it, so inside the container:
```
[root@6c37882d3465 project]# cd /tmp/tmp.iBdPtyDHa7/
[root@6c37882d3465 tmp.iBdPtyDHa7]# source venv/bin/activate
(venv) [root@6c37882d3465 tmp.iBdPtyDHa7]# python -m libcst.tool codemod remove_unused_imports.RemoveUnusedImportsCommand /project/libcst/_nodes
Calculating full-repo metadata...
Executing codemod...
Segmentation fault (core dumped)
```
Nice! Time to roll up the sleeves.
## Debugging the Segmentation fault
Took me a while to find the core files are actually dumped on _the Windows host_ in `~/AppData/Local/Temp/wsl-crashes` because
```sh
# sysctl kernel.core_pattern
kernel.core_pattern = |/wsl-capture-crash %t %E %p %s
```
Pointed me to [this issue](https://github.com/microsoft/WSL/issues/11997) where the above folder was casually mentioned.
After copying the core dump back to a container that has `gdb` installed, then installing the built `libcst` wheel (along with `pyyaml-ft`), I can finally trigger the crash in a debugger:
```
(venv) root@baf1deba1eec:/opt# gdb --args python -c 'import libcst; libcst.parse_module("")'
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
[...]
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run
Starting program: /opt/venv/bin/python -c import\ libcst\;\ libcst.parse_module\(\"\"\)
[...]
Program received signal SIGSEGV, Segmentation fault.
__strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:229
229 ../sysdeps/i386/i686/multiarch/strcmp-sse4.S: No such file or directory.
(gdb) bt
#0 __strcmp_sse4_2 () at ../sysdeps/i386/i686/multiarch/strcmp-sse4.S:229
#1 0x082181f3 in ?? ()
#2 0x08114fa7 in ?? ()
#3 0xf6fa25a5 in pyo3::sync::GILOnceCell<T>::init () from /opt/venv/lib/python3.14/site-packages/libcst/native.cpython-314-i386-linux-gnu.so
#4 0xf7139752 in pyo3::impl_::pymodule::ModuleDef::make_module () from /opt/venv/lib/python3.14/site-packages/libcst/native.cpython-314-i386-linux-gnu.so
#5 0xf71292ed in PyInit_native () from /opt/venv/lib/python3.14/site-packages/libcst/native.cpython-314-i386-linux-gnu.so
[...]
```
So looks like the crash is happening in pyo3. This was enough to get me to search the pyo3 issue tracker for [`strcmp`](https://github.com/PyO3/pyo3/issues?q=is%3Aissue%20state%3Aclosed%20strcmp), which led me to [Issues when building Python libraries with Python 3.14.0b2 and PyO3 0.25 on i686 architecture](https://github.com/PyO3/pyo3/issues/5175) merely two hours after the fix was released in PyO3 0.25.1.
Indeed, bumping the pyo3 version gets all the [wheel tests to green](https://github.com/Instagram/LibCST/pull/1359) 🥳