Test suite performance: Build wheels to go faster
Posted 24 May 2023 in performance, python, and testingPython test suites on Windows typically run very slowly on my computer. For one of my projects, though, I noticed that setting up the test environment took significantly longer than executing the tests:
py311: OK (12.19=setup[8.86]+cmd[3.33] seconds)
py310: OK (9.66=setup[7.19]+cmd[2.47] seconds)
py39: OK (12.17=setup[8.75]+cmd[3.42] seconds)
py38: OK (12.05=setup[8.84]+cmd[3.20] seconds)
This post is broken up into two sections:
- How I investigated slow setup times
- How to configure tox to avoid slow setup times
Investigating slow setup times
Here's an example of the output I was seeing (paths edited for brevity):
listparser> tox -e py311
.pkg: _optional_hooks> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
.pkg: get_requires_for_build_sdist> python ...pyproject_api\_backend.py True poetry.core.masonry.api
.pkg: prepare_metadata_for_build_wheel> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
.pkg: build_sdist> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
py311: install_package> python -I -m pip install --force-reinstall --no-deps listparser-0.19.tar.gz
py311: commands[0]> .tox\py311\Scripts\python.exe -W error -m pytest
============================== test session starts ===============================
platform win32 -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0
cachedir: .tox\py311\.pytest_cache
Using --randomly-seed=3402792870
rootdir: listparser
plugins: randomly-3.12.0
collected 186 items
tests\test_super_dict.py .. [ 1%]
tests\test_dates.py ....................................................... [ 30%]
........ [ 34%]
tests\test_xml.py ......................................................... [ 65%]
............................................................. [ 98%]
tests\test_http.py .ss [100%]
========================= 184 passed, 2 skipped in 1.08s =========================
.pkg: _exit> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
py311: OK (10.14=setup[8.44]+cmd[1.70] seconds)
congratulations :) (10.31 seconds)
The penultimate line caught my attention: it took 8.44 seconds to setup the Python 3.11 test environment but only 1.70 seconds to execute the tests.
Note that lines starting with .pkg: were part of tox's build phase; lines starting with py311: were part of tox's environment-specific setup phase.
To diagnose this, I added -v to the tox invocation, to increase the verbosity of the output. Only the py311: lines are shown, because they're the only thing relevant here.
listparser> tox -ve py311
... snip ...
py311: install_package> python -I -m pip install --force-reinstall --no-deps listparser-0.19.tar.gz
py311: exit 0 (9.66 seconds) listparser> python -I -m pip install --force-reinstall --no-deps listparser-0.19.tar.gz pid=8268
10 seconds to install a tarball is ridiculous, but this wasn't enough information. As is often the case, you can add -v more than once to get more output, though.
...\listparser> tox -vve py311
... snip ...
py311: 1040 W install_package> python -I -m pip install --force-reinstall --no-deps listparser-0.19.tar.gz [tox\tox_env\api.py:428]
Processing listparser-0.19.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: listparser
Building wheel for listparser (pyproject.toml): started
Building wheel for listparser (pyproject.toml): finished with status 'done'
Created wheel for listparser: filename=listparser-0.19-py3-none-any.whl size=14172 sha256=...
Stored in directory: ...\wheels\b7\00\cc\bf533892c0dafa6904f5e9ee5ee9401e158fd0a23a9567f1fd
Successfully built listparser
Installing collected packages: listparser
Attempting uninstall: listparser
Found existing installation: listparser 0.19
Uninstalling listparser-0.19:
Successfully uninstalled listparser-0.19
Successfully installed listparser-0.19
py311: 10801 I exit 0 (9.76 seconds) listparser> python -I -m pip install --force-reinstall --no-deps listparser-0.19.tar.gz pid=14112 [tox\execute\api.py:275]
The first line showed that tox was about to install the .tar.gz file. The last line showed that the installation took 9.76 seconds. All the other output showed that pip was converting the .tar.gz file to a wheel and then installing the newly-created wheel.
Worse, pip was converting the tarball to a wheel for every test environment!
Configuring tox to avoid repackaging
My guess was that if I could configure tox to build a wheel instead of a tarball, pip could install that without doing a conversion.
I searched tox's documentation and found the "package" configuration directive:
option can be one of wheel, sdist, editable, editable-legacy, skip, or external
Perfect! I added the following to my tox configuration:
[testenv]
package = wheel
When I re-ran the entire test suite, however, I found that tox was packaging the wheel for each test environment. This became obvious because, instead of a non-descript .pkg: line, there are now variants of that line for each version of Python:
.pkg-cpython311: build_wheel> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
.pkg-cpython310: build_wheel> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
.pkg-cpython39: build_wheel> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
.pkg-cpython38: build_wheel> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
.pkg-pypy39: build_wheel> python ...\pyproject_api\_backend.py True poetry.core.masonry.api
I found the answer in tox's documentation for the "wheel_build_env" configuration directive:
default value of <package_env>-<python-flavor-lowercase><python-version-no-dot>
If you have a wheel that can be reused across multiple Python versions set this value to the same across them (to avoid building a new wheel for each one of them).
I updated my tox configuration again:
[testenv]
package = wheel
wheel_build_env = build_wheel
With this configuration, the project was packaged once by tox and installed quickly by pip in each test environment. Here's the abbreviated output, showing much-improved setup times:
py311: OK (4.89=setup[1.31]+cmd[3.58] seconds)
py310: OK (5.80=setup[2.12]+cmd[3.67] seconds)
py39: OK (5.03=setup[1.34]+cmd[3.69] seconds)
py38: OK (4.92=setup[2.26]+cmd[2.66] seconds)
NOTE: This post was written when Python 3.11.3 and Tox 4.5.1 were current versions.