refactor(tests): the last Python harness in this repository is gone #219
Loading…
Reference in a new issue
No description provided.
Delete branch "sean/shell-tests"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The five tests/*.py files and the Python heredoc in scripts/check-recipes are shell with jq now. Every assertion is kept and each is proved to fail on a deliberate break: 53 negative controls. Two tests imported the recipe and called a function no command reached, so verify-bootc.py gains check-facts (judges supplied facts, observes nothing, never called by the recipe) and schema-payload gains a verify mode (what apply reads before it runs an installer). Writing it one case at a time made make check twice as slow, because the subject is a Python program and a shell test pays one interpreter start per case; each test now builds a directory per case and runs the cases in groups, which brings make check from 4.39 s back to 2.91 s against 2.21 s on trunk. https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76
The five `tests/*.py` files and the Python heredoc in `scripts/check-recipes` are shell with `jq` now. Every assertion the Python carried is kept, and each is proved to fail on a deliberate break: 53 negative controls in all. Two of the five could not be converted as they stood. They imported the recipe and called a function no command reached: * `verify-bootc.py verify` is reachable only through `installed()`, which reads `/usr/lib/modules`, `rpm -qa`, `lsinitrd` and `/etc/systemd` -- the running system. `check-facts` takes the same four documents from files. It observes nothing, and the recipe never calls it, which is said beside it. * `schema-payload` had `pack` and `apply` but no `verify`, and half of that test's cases call it. `verify` is what `apply` reads before it runs an installer, so it is now a mode of its own; `dist` became optional and the existing four-argument callers are untouched. WRITING IT ONE CASE AT A TIME MADE `make check` TWICE AS SLOW. The subject is a Python program, so a shell test pays one interpreter start per case where an in-process test paid one for all of them -- the same compounding the no-Python rule exists to remove, moved from the harness to the subject. Measured on a quiet machine, best of four: | test | as Python | one case at a time | in groups | |---|---|---|---| | rpg-schema-payload | 193 ms | 1404 ms | 281 ms | | linux-bootc | 73 ms | 1317 ms | 406 ms | | linux-artifact | 125 ms | 647 ms | 282 ms | | rpg-recovery | 175 ms | -- | 176 ms | | rpg-native-release | 150 ms | -- | 103 ms | | **`make check`** | **2.21 s** | **4.39 s** | **2.91 s** | Each test that drives a Python program now builds a directory per case, starts the cases in groups, and judges the recorded verdicts after. The remaining 0.7 s is the price of driving the real command line instead of an import, and AGENTS.md records the shape so the next case keeps it. One fixture is a frozen 117-byte archive in base64. `zip` and `jar` both normalise a traversal path away, so an archive holding `../escape` cannot be made by an ordinary tool -- which is the property under test. The test checks the fixture's digest and finds the name in its bytes, so it needs no archive tool of its own. The `../escape` case that `schema-payload` refuses at pack time is kept. The forged-archive case is built by packing a different file list rather than by editing an archive, which covers a member added and a member removed. Claude-Session: https://claude.ai/code/session_01KZoQin34jeyt6nDGqvJA76