sourcerer compare names the check behind an unsupported_layout refusal #794

Manually merged
binjovi-bot merged 1 commit from lcmp into trunk 2026-11-13 14:39:06 +00:00
Owner

sourcerer compare names the check behind an unsupported_layout refusal

Lane Lcmp. The change touches none of rpg_source.zig, rpg_eval.zig, rpg_statements.zig or the golden
test. It adds a site to each refusal of the comparer and nothing else: it accepts no new layout and moves no
verdict. The reason stays unsupported_layout, byte for byte.

What changes

Every return of UnsupportedComparisonLayout (ten, nine in compare_ile.zig and one in app/compare.zig)
goes through comparison.refuse(site, value), which records the check and the value it read and returns the
same error. app/compare.zig reads the record once per side into the new Evidence.refusal and clears it
before each side starts. Three compound checks are split so that one token names one check, with the same
order and the same outcome: the chunk, construction and pointer counts of a static frame (limit 4096 each),
and the registered count and the handler count of the exports.

site check (compare_ile.zig unless named) value
receipt_type app/compare.zig load: the receipt is not a PGM none
object_kind prepare: the saved object is not a program the kind word
layout_kind prepare: the image is not a program layout none
module_count prepare: the module table does not hold one module the count
header_78_size, header_80_size creationData: size of the table at module header +0x78 / +0x80 outside 16 B to 1 MiB the size
constant_count staticData: the activation header does not count one constant area the count
static_frame_count staticData: the activation header (+56) does not count one static frame the count
static_frame_size staticData: frame size 0 or above 16 MiB the size
static_chunk_count, static_construction_count, static_pointer_count staticData: more than 4096 rows the count
registered_count, handler_count exportData: registered count above the limit, or handlers differ from it the count

Output. The first line is unchanged (comparison refused: unsupported_layout, exit 3). The text form adds one
line per side that carries a site, after it and before the different component: lines, in a form that starts
with neither comparison nor different component: (roundtrip.sh and same-side.sh key on those):

refusal reference site=static_frame_count value=2
refusal twin site=static_frame_count value=2

The JSON verdict (schema sourcerer.comparison.v1) adds reference.refusal and twin.refusal, present only
on a side that carries a site: {"site": "...", "value": N or null} (a reader sees null for an absent key).
A report without a site is byte for byte what it was: the three golden structural-comparison.json files of
tests/cli/rpg-corpus.sh pin it, and an always-present null key failed that check on the first ship.
result, reason, components, counts, different_components and the exit code are as before.
docs/COMPARISON.md lists the report fields and is now short of this key and this line; it is not edited here
(coordinator).

staticData is pub so that a test without the decompiler can reach it. compare_site_test.zig
(registered in src/root.zig) asserts: the three refused two-frame twins name static_frame_count with the
value read straight from the fixture, the eight matching twins name no site, every check of the static data
names itself and its value on a mutated copy, prepare names the object kind, the report keeps the reason and
carries the site in the text and in the JSON, a recorded refusal does not move to a side that failed for
another reason, and a scan of the sources proves that no return error.UnsupportedComparisonLayout is left
and that every site token is used by exactly one check. Red: with the static_frame_count call replaced by
the bare error, three of the seven tests fail.

No verdict moved

The twins: every committed *.savf under src/testdata (1718; 1669 programs) was dumped with the base
binary (trunk 84eeba13) and compared with the base and the head binary, each program with itself (1669) and
each observable twin with its stripped twin (749): 2418 comparisons. For each, the exit code, the text without
the new lines, and the JSON (without the two refusal keys of a refused side, the only ones there are) are
byte-identical between the two binaries. The
six Async captures and the round-trip objects give the same verdicts as before (refused, exit 3).

Measurements (not shipped)

The five programs and asyncprom (it does not compile, so there is no rebuilt object), the original objects
(runs/pub-l10-async-<id>-11*/cases/1/ref/dump/<NAME>.PGM) and the captured observable twins, each compared
with itself and with its rebuilt or stripped twin: every comparison refuses at static_frame_count,
value 2
, on the reference and on the twin. Nothing else fires first; no other site fires on any of them.

program original activation +56 / +60 / +68 frame 1 (thread-local) frame 2 (plain)
asyncage PC54196B4 2 / 1 / 1 4384 B, 6 chunks, 1 construction (offset 3536, target 5944) 82112 B, 1 chunk of 176 B at 0, header +8 = 1, +16 record 0, 0x14000, 0xb0, 0
asyncdemo P55132AAD 2 / 1 / 1 4480 B, 6 chunks, 1 construction (3632, 6224) 82064 B, 1 chunk of 128 B, +8 = 1, record 0, 0x14000, 0x80, 0
asyncbg P292F548B 2 / 1 / 1 4480 B, 1 construction (3632, 6128) 82064 B, same shape
asynccncl P42C20EFE 2 / 1 / 1 4480 B, 1 construction (3632, 6168) 82064 B, same shape
asynctime P650911A0 2 / 1 / 1 4448 B, 1 construction (3600, 6296) 82064 B, same shape
asyncprom P389F4503 2 / 1 / 1 4448 B, 1 construction (3600, 6392) 82064 B, same shape

Frame 1 has flag word 0x20000 and frame 2 0x10000 (header +4). The frame header table is at activation +48,
80 bytes per header; the two relocation rows that fill the frame slots are type 0x0c (ordinal 1) and type 0
(ordinal 2), as in the committed two-frame twins. The observable and the stripped twin of a program have the
same values. 0x14000 = 4096 pointers + 4096 int(10): gTask and gGen of the Async source (dcl-c ASYNC_MAX_TASKS 4096, both static(*allthread)).

The THREAD twins. Of the 14 committed twin directories whose source has thread(*concurrent) (28 objects),
22 compare (match against themselves) with no site, and exactly 6 refuse, all at static_frame_count,
value 2: the observable and the stripped object of lext-static-all-c, lext-allthread-local-c and
lext-allthread-local-noinz-c. The six twins that match have one static frame (+56 = 1): lext-call-void-c,
-call-const-c, -call-value-c, -call-ref-c, -call-nested-c, -call-ret-c. Across all 2418 comparisons
the site static_frame_count fires only on the three refused lext twins; no other site fires on any twin.
So thread mode is not the cause: a second, plain static frame is (static(*allthread), which the matching
twins do not have).

Design for change 2 (what the comparer would read; not shipped)

Four places read the static frames as if there were one. All read the frame count at activation +56 and the
header table at +48 (80 bytes per header): compare_ile.staticData (count != 1, then one header),
compare_ile.relocationData (a frame row must have ordinal 1 at +8, header+56 == 1, and a type 0x0c row
requires +60 == 0 and +68 == 1), constant_references.initialPointers (count != 1, one header), and
constant_references.tables, which proves the tables of the module static header at +8, +16, +32, +48 and +64
(the header that starts with 0x0100000000000000) and not the one at +24.

Experiment, in a scratch copy of the tree that is never shipped (runs/lcmp-scratch/exp), measured by the
verdicts of sourcerer compare on the six programs, the round trips and the 2418 committed comparisons. The
edits are cumulative; each step says what the build before it refused with:

  1. staticData loops over the count headers (the same checks and hashes per frame) and hashes the 16-byte
    record behind header +16 instead of the address (inferred necessary, because the address differs between
    an observable and its stripped twin; never seen failing), and relocationData takes a frame ordinal in
    1..header+56 and a type 0x0c row with +68 >= 1 (the +60 == 0 rule dropped, not replaced). Refused:
    ambiguous_reference at the construction check of frame 1. The row word is 0x81000001 where the 16
    constructions of lext-serialize-c and lext-call-void-p are 0x80000001, and the target is beyond the
    size of frame 1 (5944 against 4384) but inside frame 2 (82112).
  2. The target bound is skipped when the row word is not 0x80000001. Refused: unsupported_relocation at
    initialPointers (count != 1).
  3. initialPointers loops over the frames. Each program matches itself and differs from its own observable or
    stripped twin in constant_area only: the area (24064 B in each of the six) holds one 8-byte address of
    itself that no reference proves (constant_area_unproved_pointers is 1 in each, 0 in the single-frame
    twins).
  4. tables also reads the table at statics + 24, stride 16. That word is its pointer: the target has length
    0x20 and count 1, the format table() checks, and its row is the descriptor 0, 0x14000, 0xb0, 0 that
    header +16 of frame 2 names (read on asyncage; the committed twins have a null +24, since they matched
    before this step). The six originals match themselves, each observable matches its stripped twin with no
    different component, and constant_area_unproved_pointers is 0.

With the four steps, the five programs and asyncprom compare, and the three refused lext twins compare
(match, self and observable against stripped). Of the 2418 committed comparisons only those 9 rows move
(refused to match); the other 2409 keep result, reason and components. The round trips then give a
verdict: each original against its rebuilt twin is different (in instructions, control_flow,
creation_instructions, symbols, types, literals, constant_area, initialization, imports, exports and
relocations, and on asyncage in attributes too) where it was refused.

Proposed for change 2 and not exercised: bound the frame offset of a relocation row and the target of a
construction by the size of the frame that the row (ordinal) or the row word names (0x81000001 against
0x80000001: one value seen), instead of skipping the bound; require +60 and +68 to equal the number of rows of
type 0 and of type 0x0c, each ordinal used once; cap the frame count (2 observed). Inferred, not seen failing:
the hash of the record behind header +16.

Request

~/data/sourcerer-loop/requests/lcmp/frames-two/ (12 cases, README names the question): from which size a
zero-initialised static(*allthread) array gives frame 2 the +8 = 1 flag, the record and the table at +24;
inz against no inz; the same array without static(*allthread); and a thread-local pointer start value of
an allthread field, of a thread-local field and of the big array, to read the row word 0x81000001. Not run (no
host job in this lane). The reader is a throwaway test of the lane (in the data tree,
runs/lcmp-scratch/lcmp_probe_scratch_test.zig.txt), not shipped.

Gate

== a. development emits (dev-hash)
p1 same
p2 same
p3 same
p4 same
p5 same
== b. L10-test direction
L10-test unchanged
== c. committed twins (emit-bytes-diff)
emit-bytes-diff: identical
== d. captures (capture-emit-diff)
capture-emit-diff: 6 cases, 0 changed, 0 with no failure line in the head emit
== e. source agreement (agree)
suspect        base=75 head=75
conflict       base=0 head=0
params_wrong   base=14 head=14
decl_missing   base=81 head=81
decl_extra     base=25 head=25
type_unmarked  base=45 head=45
gate: PASS
# `sourcerer compare` names the check behind an unsupported_layout refusal Lane Lcmp. The change touches none of `rpg_source.zig`, `rpg_eval.zig`, `rpg_statements.zig` or the golden test. It adds a site to each refusal of the comparer and nothing else: it accepts no new layout and moves no verdict. The reason stays `unsupported_layout`, byte for byte. ## What changes Every return of `UnsupportedComparisonLayout` (ten, nine in `compare_ile.zig` and one in `app/compare.zig`) goes through `comparison.refuse(site, value)`, which records the check and the value it read and returns the same error. `app/compare.zig` reads the record once per side into the new `Evidence.refusal` and clears it before each side starts. Three compound checks are split so that one token names one check, with the same order and the same outcome: the chunk, construction and pointer counts of a static frame (limit 4096 each), and the registered count and the handler count of the exports. | site | check (`compare_ile.zig` unless named) | value | |---|---|---| | `receipt_type` | `app/compare.zig load`: the receipt is not a `PGM` | none | | `object_kind` | `prepare`: the saved object is not a program | the kind word | | `layout_kind` | `prepare`: the image is not a program layout | none | | `module_count` | `prepare`: the module table does not hold one module | the count | | `header_78_size`, `header_80_size` | `creationData`: size of the table at module header +0x78 / +0x80 outside 16 B to 1 MiB | the size | | `constant_count` | `staticData`: the activation header does not count one constant area | the count | | `static_frame_count` | `staticData`: the activation header (+56) does not count one static frame | the count | | `static_frame_size` | `staticData`: frame size 0 or above 16 MiB | the size | | `static_chunk_count`, `static_construction_count`, `static_pointer_count` | `staticData`: more than 4096 rows | the count | | `registered_count`, `handler_count` | `exportData`: registered count above the limit, or handlers differ from it | the count | Output. The first line is unchanged (`comparison refused: unsupported_layout`, exit 3). The text form adds one line per side that carries a site, after it and before the `different component:` lines, in a form that starts with neither `comparison ` nor `different component: ` (`roundtrip.sh` and `same-side.sh` key on those): refusal reference site=static_frame_count value=2 refusal twin site=static_frame_count value=2 The JSON verdict (schema `sourcerer.comparison.v1`) adds `reference.refusal` and `twin.refusal`, present only on a side that carries a site: `{"site": "...", "value": N or null}` (a reader sees `null` for an absent key). A report without a site is byte for byte what it was: the three golden `structural-comparison.json` files of `tests/cli/rpg-corpus.sh` pin it, and an always-present `null` key failed that check on the first ship. `result`, `reason`, `components`, `counts`, `different_components` and the exit code are as before. `docs/COMPARISON.md` lists the report fields and is now short of this key and this line; it is not edited here (coordinator). `staticData` is `pub` so that a test without the decompiler can reach it. `compare_site_test.zig` (registered in `src/root.zig`) asserts: the three refused two-frame twins name `static_frame_count` with the value read straight from the fixture, the eight matching twins name no site, every check of the static data names itself and its value on a mutated copy, `prepare` names the object kind, the report keeps the reason and carries the site in the text and in the JSON, a recorded refusal does not move to a side that failed for another reason, and a scan of the sources proves that no `return error.UnsupportedComparisonLayout` is left and that every site token is used by exactly one check. Red: with the `static_frame_count` call replaced by the bare error, three of the seven tests fail. ## No verdict moved The twins: every committed `*.savf` under `src/testdata` (1718; 1669 programs) was dumped with the base binary (trunk 84eeba13) and compared with the base and the head binary, each program with itself (1669) and each observable twin with its stripped twin (749): 2418 comparisons. For each, the exit code, the text without the new lines, and the JSON (without the two `refusal` keys of a refused side, the only ones there are) are byte-identical between the two binaries. The six Async captures and the round-trip objects give the same verdicts as before (refused, exit 3). ## Measurements (not shipped) The five programs and `asyncprom` (it does not compile, so there is no rebuilt object), the original objects (`runs/pub-l10-async-<id>-11*/cases/1/ref/dump/<NAME>.PGM`) and the captured observable twins, each compared with itself and with its rebuilt or stripped twin: every comparison refuses at **`static_frame_count`, value 2**, on the reference and on the twin. Nothing else fires first; no other site fires on any of them. | program | original | activation +56 / +60 / +68 | frame 1 (thread-local) | frame 2 (plain) | |---|---|---|---|---| | asyncage | PC54196B4 | 2 / 1 / 1 | 4384 B, 6 chunks, 1 construction (offset 3536, target 5944) | 82112 B, 1 chunk of 176 B at 0, header +8 = 1, +16 record `0, 0x14000, 0xb0, 0` | | asyncdemo | P55132AAD | 2 / 1 / 1 | 4480 B, 6 chunks, 1 construction (3632, 6224) | 82064 B, 1 chunk of 128 B, +8 = 1, record `0, 0x14000, 0x80, 0` | | asyncbg | P292F548B | 2 / 1 / 1 | 4480 B, 1 construction (3632, 6128) | 82064 B, same shape | | asynccncl | P42C20EFE | 2 / 1 / 1 | 4480 B, 1 construction (3632, 6168) | 82064 B, same shape | | asynctime | P650911A0 | 2 / 1 / 1 | 4448 B, 1 construction (3600, 6296) | 82064 B, same shape | | asyncprom | P389F4503 | 2 / 1 / 1 | 4448 B, 1 construction (3600, 6392) | 82064 B, same shape | Frame 1 has flag word 0x20000 and frame 2 0x10000 (header +4). The frame header table is at activation +48, 80 bytes per header; the two relocation rows that fill the frame slots are type 0x0c (ordinal 1) and type 0 (ordinal 2), as in the committed two-frame twins. The observable and the stripped twin of a program have the same values. 0x14000 = 4096 pointers + 4096 int(10): `gTask` and `gGen` of the Async source (`dcl-c ASYNC_MAX_TASKS 4096`, both `static(*allthread)`). The THREAD twins. Of the 14 committed twin directories whose source has `thread(*concurrent)` (28 objects), 22 compare (`match` against themselves) with no site, and exactly 6 refuse, all at `static_frame_count`, value 2: the observable and the stripped object of `lext-static-all-c`, `lext-allthread-local-c` and `lext-allthread-local-noinz-c`. The six twins that match have one static frame (+56 = 1): `lext-call-void-c`, `-call-const-c`, `-call-value-c`, `-call-ref-c`, `-call-nested-c`, `-call-ret-c`. Across all 2418 comparisons the site `static_frame_count` fires only on the three refused lext twins; no other site fires on any twin. So thread mode is not the cause: a second, plain static frame is (`static(*allthread)`, which the matching twins do not have). ## Design for change 2 (what the comparer would read; not shipped) Four places read the static frames as if there were one. All read the frame count at activation +56 and the header table at +48 (80 bytes per header): `compare_ile.staticData` (`count != 1`, then one header), `compare_ile.relocationData` (a frame row must have ordinal 1 at +8, `header+56 == 1`, and a type 0x0c row requires +60 == 0 and +68 == 1), `constant_references.initialPointers` (`count != 1`, one header), and `constant_references.tables`, which proves the tables of the module static header at +8, +16, +32, +48 and +64 (the header that starts with 0x0100000000000000) and not the one at +24. Experiment, in a scratch copy of the tree that is never shipped (`runs/lcmp-scratch/exp`), measured by the verdicts of `sourcerer compare` on the six programs, the round trips and the 2418 committed comparisons. The edits are cumulative; each step says what the build before it refused with: 1. `staticData` loops over the `count` headers (the same checks and hashes per frame) and hashes the 16-byte record behind header +16 instead of the address (inferred necessary, because the address differs between an observable and its stripped twin; never seen failing), and `relocationData` takes a frame ordinal in 1..`header+56` and a type 0x0c row with +68 >= 1 (the +60 == 0 rule dropped, not replaced). Refused: `ambiguous_reference` at the construction check of frame 1. The row word is 0x81000001 where the 16 constructions of `lext-serialize-c` and `lext-call-void-p` are 0x80000001, and the target is beyond the size of frame 1 (5944 against 4384) but inside frame 2 (82112). 2. The target bound is skipped when the row word is not 0x80000001. Refused: `unsupported_relocation` at `initialPointers` (`count != 1`). 3. `initialPointers` loops over the frames. Each program matches itself and differs from its own observable or stripped twin in `constant_area` only: the area (24064 B in each of the six) holds one 8-byte address of itself that no reference proves (`constant_area_unproved_pointers` is 1 in each, 0 in the single-frame twins). 4. `tables` also reads the table at `statics + 24`, stride 16. That word is its pointer: the target has length 0x20 and count 1, the format `table()` checks, and its row is the descriptor `0, 0x14000, 0xb0, 0` that header +16 of frame 2 names (read on asyncage; the committed twins have a null +24, since they matched before this step). The six originals match themselves, each observable matches its stripped twin with no different component, and `constant_area_unproved_pointers` is 0. With the four steps, the five programs and `asyncprom` compare, and the three refused lext twins compare (`match`, self and observable against stripped). Of the 2418 committed comparisons only those 9 rows move (`refused` to `match`); the other 2409 keep result, reason and components. The round trips then give a verdict: each original against its rebuilt twin is `different` (in instructions, control_flow, creation_instructions, symbols, types, literals, constant_area, initialization, imports, exports and relocations, and on asyncage in attributes too) where it was `refused`. Proposed for change 2 and not exercised: bound the frame offset of a relocation row and the target of a construction by the size of the frame that the row (ordinal) or the row word names (0x81000001 against 0x80000001: one value seen), instead of skipping the bound; require +60 and +68 to equal the number of rows of type 0 and of type 0x0c, each ordinal used once; cap the frame count (2 observed). Inferred, not seen failing: the hash of the record behind header +16. ## Request `~/data/sourcerer-loop/requests/lcmp/frames-two/` (12 cases, README names the question): from which size a zero-initialised `static(*allthread)` array gives frame 2 the +8 = 1 flag, the record and the table at +24; `inz` against no `inz`; the same array without `static(*allthread)`; and a thread-local pointer start value of an allthread field, of a thread-local field and of the big array, to read the row word 0x81000001. Not run (no host job in this lane). The reader is a throwaway test of the lane (in the data tree, `runs/lcmp-scratch/lcmp_probe_scratch_test.zig.txt`), not shipped. ## Gate ``` == a. development emits (dev-hash) p1 same p2 same p3 same p4 same p5 same == b. L10-test direction L10-test unchanged == c. committed twins (emit-bytes-diff) emit-bytes-diff: identical == d. captures (capture-emit-diff) capture-emit-diff: 6 cases, 0 changed, 0 with no failure line in the head emit == e. source agreement (agree) suspect base=75 head=75 conflict base=0 head=0 params_wrong base=14 head=14 decl_missing base=81 head=81 decl_extra base=25 head=25 type_unmarked base=45 head=45 gate: PASS ```
Name the check behind an unsupported_layout refusal of sourcerer compare
Some checks reported errors
binjovi/ci A newer source or plan superseded this Build
86666a077b
Every return of UnsupportedComparisonLayout goes through comparison.refuse, which records a site token and the value the check read. The reason stays unsupported_layout; the text form adds one refusal line per side and the JSON verdict adds a refusal key to a side that has a site. No layout is accepted and no verdict moves (2418 committed comparisons identical between the base and the head binary).
sean force-pushed lcmp from 86666a077b
Some checks reported errors
binjovi/ci A newer source or plan superseded this Build
to c4b33816e6
All checks were successful
binjovi/ci Binjovi completed the frozen plan
2026-11-13 14:26:26 +00:00
Compare
binjovi-bot manually merged commit f3c6575589 into trunk 2026-11-13 14:39:06 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/sourcerer!794
No description provided.