sourcerer compare reads the two static frames of a thread(*concurrent) module with static(*allthread) fields #795

Manually merged
binjovi-bot merged 1 commit from lcmp2 into trunk 2026-11-13 17:07:29 +00:00
Owner

sourcerer compare reads the two static frames of a thread(*concurrent) module with static(*allthread) fields

Lane Lcmp, change 2, stacked on lcmp (86666a07, the sites of an unsupported_layout refusal). The change
touches none of rpg_source.zig, rpg_eval.zig, rpg_statements.zig or the golden test. It accepts one new
layout, a module of two static frames, and nothing else: the three refused lext twins and the six public
Async programs now compare. A layout that is not in a public object is still refused, with a site.

What changes

The comparer assumed one static frame in four places. All four read the frame count at activation +56 and the
frame headers (80 bytes each, one after the other) from the pointer at +48.

  • compare_ile.staticData reads every frame (staticFrame). At most two frames (max_static_frames): the
    thread-local frame (header +4 = 0x20000), then the plain frame (0x10000). The size of every frame is read
    first, because a construction row names the frame of its target.
  • In a module of two frames, header +8 is 1 and +16 names a 16-byte record exactly for the second frame with a
    zero tail (the Async programs: the plain frame of 82064 to 82112 bytes, the record 0, 0x14000, 0xb0 or 0x80, 0); the first frame has none. The record is hashed instead of the pointer, which differs between an
    observable and a stripped twin. The second frame has no construction and no pointer initializer in any public
    object: it has none here (static_second_frame_rows).
  • A construction row of a module with two frames has the word 0x80000001 (a target in the first frame) or
    0x81000001 (the second): the thread-local frame of the Async programs points into the plain one. The target
    is bounded by the size of the frame that the word names; the place of the pointer by its own frame.
  • compare_ile.relocationData: with two frames, the row of type 0x0c names frame 1 and the row of type 0
    names frame 2, once each, and the header counts +60 and +68 are 1 each. The offset of a row is bounded by
    the frame it names.
  • constant_references.initialPointers reads the pointer initializers of every frame, and
    constant_references.tables proves the table at statics + 24 (16-byte rows: the records of the frames with
    a zero tail), at most as many rows as frames. Without it one 8-byte address in the constant area stayed exact
    and an observable twin differed from its stripped twin in constant_area only.

Fail-closed: more than two frames (static_frame_count), two frames that are not thread-local then plain
(static_frame_flags), a header +8 and +16 that disagree or a record on the first frame
(static_frame_record), pointer constructions or initializers in the second frame
(static_second_frame_rows), a construction row word other than the two (static_construction_word), a
target beyond its frame (static_construction_target), a frame size outside 1 byte to 16 MiB
(static_frame_size, as before) refuse with their site;
a relocation row, count or ordinal that does not fit the frames is InvalidRelocation
(unsupported_relocation), and a table at +24 with more rows than frames is UnsupportedRelocation. A module
of one frame is read as before: its row words, header bytes (+8 and +16 are hashed raw) and relocation rows
are not newly checked, so that no verdict of a single-frame object moves. The five new sites are in
docs/COMPARISON.md, which also carries the refusal line and the JSON key of change 1 and a section
"Static frames".

Tests

Red first, in compare_site_test.zig (no decompiler, run by ship-unit.sh): with the sites added and
staticData as it was, 6 of 11 failed (the three refused twins still refused, the third frame, the frame
flags, the record and the construction rows named no site). Green after the change. In the Build image, with
compare_ile.zig and constant_references.zig of change 1 under the tests of this change, 10 tests fail: the
four of compare_frames_test.zig and six of compare_site_test.zig.

  • the three lext two-frame twins: no refusal, and the hash of the stripped object equals the hash of the
    observable one (the addresses differ); another program is another hash;
  • a third frame, flags 0x30000 or 0x20000 on the second frame, a size of 0: each its site and value;
  • the record behind +16: equal hash for two objects with other addresses, another tail is another hash,
    +8 = 2, +8 = 0 with a record, +8 = 1 without a record, +8 = 1 or a record on the first frame:
    static_frame_record; a module of one frame with +8 = 2 and a +16 pointer is not refused (its header is
    hashed raw, as before);
  • constructions or pointer initializers in the second frame: static_second_frame_rows;
  • a construction row (written over the first chunk of a copy): 0x80000001 and 0x81000001 are read and bounded
    (target 3087 / 63 accepted, 3088 / 64 refused with the target), and five other words are refused with the
    word.

compare_frames_test.zig collects whole objects, so it needs the decompiler. build.zig runs it as a root of
its own (ship-unit.sh leaves a file that build.zig names to the full check, as dump_test.zig):
the observable and the stripped twin of each of the three programs match, each matches itself, and they differ
from each other and from a one-frame program; a third frame (static_frame_count), one frame where the rows
and counts say two, a thread-local row naming frame 2 or a plain row naming frame 1 or 3, +60 = 0, +68 = 2,
two rows of one type, and a row at the end of frame 2 are refused; a table at +24 that copies the frame index
table (two rows) is proved, and the twins still match, while one that copies the table of initial byte ranges
(seven rows) is UnsupportedRelocation.

Sweep of the committed twins (base = the change-1 binary)

The same 2418 comparisons as in change 1 (1718 committed *.savf, 1669 programs; each program with itself and
each observable twin with its stripped twin; dumps made with the base binary) with the change-1 binary
(runs/sourcerer-lcmp/out/bin/sourcerer) and the change-2 binary. Moved rows: 9, none other.

old new rows
refused / unsupported_layout match / none lext-static-all-c, lext-allthread-local-c, lext-allthread-local-noinz-c: the stripped and the observable object against itself (6), and the observable against the stripped twin (3)

The other 2409 rows have the same exit code, result, reason, different components, counts and hashes
(the whole JSON). No row moved from match to another verdict.

The six Async programs

Against itself and the observable against the stripped twin (captures l10-pgm-a1/*, l10-pgm-a2/*): match,
no different component, constant_area_unproved_pointers 0, all six (asyncprom included). Each original
against its rebuilt object (runs/pub-l10-async-<id>-11*/cases/1/ref/dump/<NAME>.PGM against
cases/1/capture/stripped): different for the five that compile, in instructions, control_flow,
creation_instructions, symbols, types, literals, constant_area, initialization, imports,
exports and relocations (asyncage also in attributes), and in compilation, which is reported and never
decides. Before this change they were refused. asyncprom does not compile: no rebuilt object.

What the tests do not cover

No committed object has a table at +24, a record behind header +16, or a construction row in a module of two
frames: the accept path of these three is tested on crafted copies of committed twins (a copy of the frame
index table as the table at +24, a record written over chunk data, a row written over chunk data) and, as the
real thing, by the CLI on the six Async captures (public, in the data tree, not committed).

Not done

The request requests/lcmp/frames-two is not run (no host job). Which source shapes give the second frame
its record, the table at +24 and the cross-frame row word is still read from six programs: a shape of a module
of two frames that no public object has is refused, not guessed.

Gate

== a. development emits (dev-hash)
p1 same
p2 same
p3 same
p4 same
p5 same
== b. L10-test direction
L10-test unchanged
== c. committed twins (emit-bytes-diff)
emit-bytes-diff: identical
== d. captures (capture-emit-diff)
capture-emit-diff: 30 cases, 0 changed, 0 with no failure line in the head emit
== e. source agreement (agree)
suspect        base=83 head=83
conflict       base=0 head=0
params_wrong   base=22 head=22
decl_missing   base=87 head=87
decl_extra     base=31 head=31
type_unmarked  base=46 head=46
gate: PASS
# `sourcerer compare` reads the two static frames of a thread(*concurrent) module with static(*allthread) fields Lane Lcmp, change 2, stacked on `lcmp` (86666a07, the sites of an `unsupported_layout` refusal). The change touches none of `rpg_source.zig`, `rpg_eval.zig`, `rpg_statements.zig` or the golden test. It accepts one new layout, a module of two static frames, and nothing else: the three refused `lext` twins and the six public Async programs now compare. A layout that is not in a public object is still refused, with a site. ## What changes The comparer assumed one static frame in four places. All four read the frame count at activation +56 and the frame headers (80 bytes each, one after the other) from the pointer at +48. - `compare_ile.staticData` reads every frame (`staticFrame`). At most two frames (`max_static_frames`): the thread-local frame (header +4 = 0x20000), then the plain frame (0x10000). The size of every frame is read first, because a construction row names the frame of its target. - In a module of two frames, header +8 is 1 and +16 names a 16-byte record exactly for the second frame with a zero tail (the Async programs: the plain frame of 82064 to 82112 bytes, the record `0, 0x14000, 0xb0 or 0x80, 0`); the first frame has none. The record is hashed instead of the pointer, which differs between an observable and a stripped twin. The second frame has no construction and no pointer initializer in any public object: it has none here (`static_second_frame_rows`). - A construction row of a module with two frames has the word 0x80000001 (a target in the first frame) or 0x81000001 (the second): the thread-local frame of the Async programs points into the plain one. The target is bounded by the size of the frame that the word names; the place of the pointer by its own frame. - `compare_ile.relocationData`: with two frames, the row of type 0x0c names frame 1 and the row of type 0 names frame 2, once each, and the header counts +60 and +68 are 1 each. The offset of a row is bounded by the frame it names. - `constant_references.initialPointers` reads the pointer initializers of every frame, and `constant_references.tables` proves the table at `statics + 24` (16-byte rows: the records of the frames with a zero tail), at most as many rows as frames. Without it one 8-byte address in the constant area stayed exact and an observable twin differed from its stripped twin in `constant_area` only. Fail-closed: more than two frames (`static_frame_count`), two frames that are not thread-local then plain (`static_frame_flags`), a header +8 and +16 that disagree or a record on the first frame (`static_frame_record`), pointer constructions or initializers in the second frame (`static_second_frame_rows`), a construction row word other than the two (`static_construction_word`), a target beyond its frame (`static_construction_target`), a frame size outside 1 byte to 16 MiB (`static_frame_size`, as before) refuse with their site; a relocation row, count or ordinal that does not fit the frames is `InvalidRelocation` (`unsupported_relocation`), and a table at +24 with more rows than frames is `UnsupportedRelocation`. A module of one frame is read as before: its row words, header bytes (+8 and +16 are hashed raw) and relocation rows are not newly checked, so that no verdict of a single-frame object moves. The five new sites are in `docs/COMPARISON.md`, which also carries the refusal line and the JSON key of change 1 and a section "Static frames". ## Tests Red first, in `compare_site_test.zig` (no decompiler, run by `ship-unit.sh`): with the sites added and `staticData` as it was, 6 of 11 failed (the three refused twins still refused, the third frame, the frame flags, the record and the construction rows named no site). Green after the change. In the Build image, with `compare_ile.zig` and `constant_references.zig` of change 1 under the tests of this change, 10 tests fail: the four of `compare_frames_test.zig` and six of `compare_site_test.zig`. - the three `lext` two-frame twins: no refusal, and the hash of the stripped object equals the hash of the observable one (the addresses differ); another program is another hash; - a third frame, flags 0x30000 or 0x20000 on the second frame, a size of 0: each its site and value; - the record behind +16: equal hash for two objects with other addresses, another tail is another hash, +8 = 2, +8 = 0 with a record, +8 = 1 without a record, +8 = 1 or a record on the first frame: `static_frame_record`; a module of one frame with +8 = 2 and a +16 pointer is not refused (its header is hashed raw, as before); - constructions or pointer initializers in the second frame: `static_second_frame_rows`; - a construction row (written over the first chunk of a copy): 0x80000001 and 0x81000001 are read and bounded (target 3087 / 63 accepted, 3088 / 64 refused with the target), and five other words are refused with the word. `compare_frames_test.zig` collects whole objects, so it needs the decompiler. `build.zig` runs it as a root of its own (`ship-unit.sh` leaves a file that `build.zig` names to the full check, as `dump_test.zig`): the observable and the stripped twin of each of the three programs match, each matches itself, and they differ from each other and from a one-frame program; a third frame (`static_frame_count`), one frame where the rows and counts say two, a thread-local row naming frame 2 or a plain row naming frame 1 or 3, +60 = 0, +68 = 2, two rows of one type, and a row at the end of frame 2 are refused; a table at +24 that copies the frame index table (two rows) is proved, and the twins still match, while one that copies the table of initial byte ranges (seven rows) is `UnsupportedRelocation`. ## Sweep of the committed twins (base = the change-1 binary) The same 2418 comparisons as in change 1 (1718 committed `*.savf`, 1669 programs; each program with itself and each observable twin with its stripped twin; dumps made with the base binary) with the change-1 binary (`runs/sourcerer-lcmp/out/bin/sourcerer`) and the change-2 binary. Moved rows: 9, none other. | old | new | rows | |---|---|---| | refused / unsupported_layout | match / none | `lext-static-all-c`, `lext-allthread-local-c`, `lext-allthread-local-noinz-c`: the stripped and the observable object against itself (6), and the observable against the stripped twin (3) | The other 2409 rows have the same exit code, result, reason, different components, counts and hashes (the whole JSON). No row moved from `match` to another verdict. ## The six Async programs Against itself and the observable against the stripped twin (captures `l10-pgm-a1/*`, `l10-pgm-a2/*`): `match`, no different component, `constant_area_unproved_pointers` 0, all six (asyncprom included). Each original against its rebuilt object (`runs/pub-l10-async-<id>-11*/cases/1/ref/dump/<NAME>.PGM` against `cases/1/capture/stripped`): `different` for the five that compile, in `instructions`, `control_flow`, `creation_instructions`, `symbols`, `types`, `literals`, `constant_area`, `initialization`, `imports`, `exports` and `relocations` (asyncage also in `attributes`), and in `compilation`, which is reported and never decides. Before this change they were `refused`. asyncprom does not compile: no rebuilt object. ## What the tests do not cover No committed object has a table at +24, a record behind header +16, or a construction row in a module of two frames: the accept path of these three is tested on crafted copies of committed twins (a copy of the frame index table as the table at +24, a record written over chunk data, a row written over chunk data) and, as the real thing, by the CLI on the six Async captures (public, in the data tree, not committed). ## Not done The request `requests/lcmp/frames-two` is not run (no host job). Which source shapes give the second frame its record, the table at +24 and the cross-frame row word is still read from six programs: a shape of a module of two frames that no public object has is refused, not guessed. ## Gate ``` == a. development emits (dev-hash) p1 same p2 same p3 same p4 same p5 same == b. L10-test direction L10-test unchanged == c. committed twins (emit-bytes-diff) emit-bytes-diff: identical == d. captures (capture-emit-diff) capture-emit-diff: 30 cases, 0 changed, 0 with no failure line in the head emit == e. source agreement (agree) suspect base=83 head=83 conflict base=0 head=0 params_wrong base=22 head=22 decl_missing base=87 head=87 decl_extra base=31 head=31 type_unmarked base=46 head=46 gate: PASS ```
Read the two static frames of a thread(*concurrent) module in sourcerer compare
Some checks reported errors
binjovi/ci A newer source or plan superseded this Build
464c6c94e8
A module with static(*allthread) fields has a thread-local frame and a plain frame. The comparison reads both: every frame header, the record behind header +16 by content, the construction rows by the frame their word names, the relocation rows by frame ordinal, the pointer initializers of every frame and the table of frame records at +24 of the static header. A third frame, a frame, record, row word or placement that no public object has, and a table with more rows than frames are refused with their site. A module of one frame reads as before.
sean force-pushed lcmp2 from 464c6c94e8
Some checks reported errors
binjovi/ci A newer source or plan superseded this Build
to a0c6ac6f71
All checks were successful
binjovi/ci Binjovi completed the frozen plan
2026-11-13 16:57:08 +00:00
Compare
binjovi-bot manually merged commit a0c6ac6f71 into trunk 2026-11-13 17:07:29 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/sourcerer!795
No description provided.