The statement readers accept thread-local static storage (touches rpg_eval.zig and rpg_source.zig) #778

Manually merged
binjovi-bot merged 3 commits from lext into trunk 2026-11-10 16:01:59 +00:00
Owner

The statement readers accept thread-local static storage (touches rpg_eval.zig and rpg_source.zig)

Lane Lext. The first measure on code written by others (the L10 development
programs: one MIT-licensed code base, l10-async-*, flattened into six
programs, counted as one) showed that recovery generalises badly. Every static
data symbol of those programs has bit 0x100 set in its flag word (byte 24 of
the symbol): 0x7000c000 is 0x7000c100, the 288-byte frame area
0xf800c000 is 0xf800c100. The sources say ctl-opt ... thread(*concurrent);
the compiler keeps the static storage of such a module in thread-local
storage. The bit is on the static data symbols of the six programs, the compiler's own
areas included, except about a dozen of each program (asyncage: 12 data symbols;
its source has nine static(*allthread) declarations; that the two are the same
symbols is an inference, the battery case static-all-c tests it). It is on no
static data symbol of the 781 public twins.

The families that stop recovery on the L10 development data

Counted on the six programs together (3,600 source statements; 1,227 recovered
before this change, 34 percent), by the stage of the first stop of each
statement that did not recover:

stage statements construct
control_flow 1,434 the blocks around other stops: if/endif (condition_boundary, closing_destination), dow, end-proc (block_owner)
type_inference 703 operation 75 at the start of every statement with a call of a procedure of the module or an import (mutexLock();, p = findTask(task);, settle(p: ...))
value_flow 155 return of a value, pthread_create, CEE4RAGE; remaining_effects after end-proc
classification 47 return *null (104), if t.resultBuf <> *null (8), dealloc (7)
call 34 a built-in call (97): t.errMsgf = 'QCPFMSG *LIBL'

944 of the 2,373 statements that did not recover carry the 30-operation call
frame (27 29 5 75 42 27 37 ...) that rpg_frames.frame proves. The frame is
refused because its areas are read by their flag word (metadata,
callInformation), and so is every other reader that names a static symbol by
its flag word (rpg_display, rpg_numeric, rpg_specs, rpg_monitor,
rpg_reset, rpg_array_element, rpg_file_ops, the file readers). The
control_flow stops follow: a block that holds an unresolved statement stays
open. One family, one rule: thread-local static storage.

The rule

crt.Symbol.flags() is the flag word without bit 0x100 of a static symbol
(thread_local_flag). The statement readers that compared word(24) of a
static symbol with a constant use it: rpg_frames (metadata,
callInformation, the header and the name), rpg_calls, rpg_numeric,
rpg_display (symbol, which rpg_display_numeric calls), rpg_specs,
rpg_spec_entry, rpg_monitor, rpg_reset, rpg_array_element,
rpg_file_ops, rpg_file_defs, rpg_file_recon, and
rpg_source_unused.sourceEnd. rpg_eval.spelledByDeclaration asks the new
rpg_source_decl.sourceField: a field in thread-local storage is a field of
the source (before, a *1 became *1.0 and a group was added, measured with the
bit set on EXPERIMENTS/ftwin14-x1). The bit does not change what a reader
proves about the bytes of a symbol: a thread-local static is a static.
Class 4 and 5 symbols (automatic, parameters) have no such bit and read as before.

The emitter does not follow. rpg_source_decl.declared keeps the raw word, so
a thread-local field still has no declaration, and ctlOpt adds
SOURCERER-FAIL THREAD(*CONCURRENT): the thread-local static storage has no emit rule
when rpg_source_decl.threadLocalStorage finds the bit (the text has no
thread(*concurrent) and no static(*allthread) yet). The readers now prove
the statements; the text of such a module stays a refusal until the emitter
writes the two options. rpg_eval.zig and rpg_source.zig are touched for these
two lines only.

Evidence and counter

Two existing tests flip each bit of the flag word of a symbol and expect a refusal
(rpg_display.zig "display descriptors refuse ...", rpg_display_numeric_test.zig
"numeric display refuses ..."). Bit 0x100 of a static symbol is no longer a
change, so on byte 26 of a static symbol they flip bit 0x200 instead.

No public twin is compiled with THREAD(*CONCURRENT), so the red test sets the
bit on the static data symbols of 25 public twins that recover without it and
asks for the same statements, text and status (src/rpg_lext_test.zig). Before
the rule, 457 of the 781 stripped twins (all the ones with a call, display,
monitor, file or loop statement) changed under the same mutation; after it, none
of the 25 does (a scan of all 781 gave 2 differences before the sourceField
change, 0 after). The gates: twins 826 groups, 804 recovered, 0 regressions, 0
newly recovered; emit 816 rows, 0 worse, 0 better.

Counter, source statements recovered on the L10 development data (six programs,
Debug build, sourcerer dump): 1,227 of 3,600 before; 1,490 of 3,449 after
(34.1 percent, 43.2 percent). Joined per statement: 428 unresolved and 208
partial statements recover, 82 unresolved move to partial; 222 statements that
were recovered get worse, and 151 rows leave the source statements. All 373
are statements whose operations are the store of the statement status
(27 0; 35 R, shown as S677 = 0): DCL-PROC lines, ENDIF and ELSE
lines (and 6 copies of tCancelThrown = *on;, whose text is as before but whose enclosing block now stays open: partial). Before, rpg_reset did not find R in a thread-local module and the
store read as an assignment, so the baseline counted them as recovered
statements with a wrong text. Now R is known and the store is bookkeeping, as in
every other module: the DCL-PROC line of a procedure that is not exported
is a declaration (151 rows left the source count; the twin
FLOW/lx-subproc shows DCL-PROC BUMP the same way), an ENDIF and an
ELSE follow the block they close (87 rows wait for the call statements inside
that stop on the status unit below), and the DCL-PROC line of an exported
procedure is refused (126 rows, see the next section). Like for like, the
baseline recovered 1,005 statements and the count is 1,490. The text of 103
newly recovered statements of asyncage was read against source.rpgle
(conditions on pointers and based fields, assignments, leave, enddo,
return): the operands are the ones of the source. Statements with a stop by
stage: control_flow 1,434 to 753, type_inference 703 to 794, classification
47 to 217 (the status stores above), value_flow 155 and call 34 to 40. The
dev emit of the six programs (savf rpgle): failures 535..576 to 421..450,
emitted 55..62 to 170..187, one THREAD(*CONCURRENT) failure line in each, no
clean output. The family removed: the flag word of a static symbol. The public
twins turned: none (the mutation twins above only keep it from regressing).

What stops next, and the battery

517 of the statements that still do not recover end their call with a
different status unit: 99 P N 0 1; 6 1; 29 A 10 7; 37 A 15 7; 27 0; 35 R; 27 T; 37 A 10 7; 6 2, where the plain module has 29 F 96 Q; 32 B 0 I2; 27 1; 7; 16 L; ...; 15 E (rpg_frames.status). It is in no public twin, so there
is no rule yet: the area A is the 429-byte static area, the field at 10 holds
a five-byte text that is copied to 15 and reset, and R (the call status) is set
to 0 without a test. Whether that holds for a call that can fail (callp(e),
monitor) is the question of the battery
requests/lext/thread-call (24 cases, with plain controls, static(*allthread),
thread(*serialize), the pointer comparison, return *null, %alloc and
dealloc). The next rule waits for its capture.

Also next: a procedure with export has symbol class 1, the class that
rpg_statements.recover takes for the cycle main procedure (is_main). 26 of
the 50 procedures of asyncage are exported. Their entry (DCL-PROC) and the
end are read as the main's, and stop with statement_role. No public twin
exports a procedure; export-* of the battery are the three cases.

Gates (ship.sh):

{
  "checks": {
    "net_statements": 0
  },
  "census": {
    "statuses": {
      "recovered": 645,
      "partial": 0,
      "unresolved": 0
    },
    "regressions": 0,
    "newly_recovered": 0
  },
  "golden": "golden floor 173 ceiling 0 <= dump 173/5/0",
  "legend": "current",
  "emit": {
    "worse": [],
    "better": 0,
    "newly_clean": 0,
    "clean_delta": 0,
    "base": 588,
    "new": 588
  },
  "twins": {
    "groups": 826,
    "recovered": 804,
    "regressions": [],
    "newly": 0
  },
  "allgroups": {
    "regressions": [],
    "newly": [],
    "worse": []
  },
  "full_check": {
    "exit": 0,
    "seconds": 856
  }
}
# The statement readers accept thread-local static storage (touches rpg_eval.zig and rpg_source.zig) Lane Lext. The first measure on code written by others (the L10 development programs: one MIT-licensed code base, `l10-async-*`, flattened into six programs, counted as one) showed that recovery generalises badly. Every static data symbol of those programs has bit 0x100 set in its flag word (byte 24 of the symbol): `0x7000c000` is `0x7000c100`, the 288-byte frame area `0xf800c000` is `0xf800c100`. The sources say `ctl-opt ... thread(*concurrent)`; the compiler keeps the static storage of such a module in thread-local storage. The bit is on the static data symbols of the six programs, the compiler's own areas included, except about a dozen of each program (asyncage: 12 data symbols; its source has nine `static(*allthread)` declarations; that the two are the same symbols is an inference, the battery case `static-all-c` tests it). It is on no static data symbol of the 781 public twins. ## The families that stop recovery on the L10 development data Counted on the six programs together (3,600 source statements; 1,227 recovered before this change, 34 percent), by the stage of the first stop of each statement that did not recover: | stage | statements | construct | |---|---|---| | control_flow | 1,434 | the blocks around other stops: `if`/`endif` (`condition_boundary`, `closing_destination`), `dow`, `end-proc` (`block_owner`) | | type_inference | 703 | operation 75 at the start of every statement with a call of a procedure of the module or an import (`mutexLock();`, `p = findTask(task);`, `settle(p: ...)`) | | value_flow | 155 | `return` of a value, `pthread_create`, `CEE4RAGE`; `remaining_effects` after `end-proc` | | classification | 47 | `return *null` (104), `if t.resultBuf <> *null` (8), `dealloc` (7) | | call | 34 | a built-in call (97): `t.errMsgf = 'QCPFMSG *LIBL'` | 944 of the 2,373 statements that did not recover carry the 30-operation call frame (`27 29 5 75 42 27 37 ...`) that `rpg_frames.frame` proves. The frame is refused because its areas are read by their flag word (`metadata`, `callInformation`), and so is every other reader that names a static symbol by its flag word (`rpg_display`, `rpg_numeric`, `rpg_specs`, `rpg_monitor`, `rpg_reset`, `rpg_array_element`, `rpg_file_ops`, the file readers). The control_flow stops follow: a block that holds an unresolved statement stays open. One family, one rule: **thread-local static storage**. ## The rule `crt.Symbol.flags()` is the flag word without bit 0x100 of a static symbol (`thread_local_flag`). The statement readers that compared `word(24)` of a static symbol with a constant use it: `rpg_frames` (`metadata`, `callInformation`, the header and the name), `rpg_calls`, `rpg_numeric`, `rpg_display` (`symbol`, which `rpg_display_numeric` calls), `rpg_specs`, `rpg_spec_entry`, `rpg_monitor`, `rpg_reset`, `rpg_array_element`, `rpg_file_ops`, `rpg_file_defs`, `rpg_file_recon`, and `rpg_source_unused.sourceEnd`. `rpg_eval.spelledByDeclaration` asks the new `rpg_source_decl.sourceField`: a field in thread-local storage is a field of the source (before, a `*1` became `*1.0` and a group was added, measured with the bit set on `EXPERIMENTS/ftwin14-x1`). The bit does not change what a reader proves about the bytes of a symbol: a thread-local static is a static. Class 4 and 5 symbols (automatic, parameters) have no such bit and read as before. The emitter does not follow. `rpg_source_decl.declared` keeps the raw word, so a thread-local field still has no declaration, and `ctlOpt` adds `SOURCERER-FAIL THREAD(*CONCURRENT): the thread-local static storage has no emit rule` when `rpg_source_decl.threadLocalStorage` finds the bit (the text has no `thread(*concurrent)` and no `static(*allthread)` yet). The readers now prove the statements; the text of such a module stays a refusal until the emitter writes the two options. `rpg_eval.zig` and `rpg_source.zig` are touched for these two lines only. ## Evidence and counter Two existing tests flip each bit of the flag word of a symbol and expect a refusal (`rpg_display.zig` "display descriptors refuse ...", `rpg_display_numeric_test.zig` "numeric display refuses ..."). Bit 0x100 of a static symbol is no longer a change, so on byte 26 of a static symbol they flip bit 0x200 instead. No public twin is compiled with THREAD(*CONCURRENT), so the red test sets the bit on the static data symbols of 25 public twins that recover without it and asks for the same statements, text and status (`src/rpg_lext_test.zig`). Before the rule, 457 of the 781 stripped twins (all the ones with a call, display, monitor, file or loop statement) changed under the same mutation; after it, none of the 25 does (a scan of all 781 gave 2 differences before the `sourceField` change, 0 after). The gates: twins 826 groups, 804 recovered, 0 regressions, 0 newly recovered; emit 816 rows, 0 worse, 0 better. Counter, source statements recovered on the L10 development data (six programs, Debug build, `sourcerer dump`): **1,227 of 3,600 before; 1,490 of 3,449 after** (34.1 percent, 43.2 percent). Joined per statement: 428 unresolved and 208 partial statements recover, 82 unresolved move to partial; 222 statements that were `recovered` get worse, and 151 rows leave the source statements. All 373 are statements whose operations are the store of the statement status (`27 0; 35 R`, shown as `S677 = 0`): `DCL-PROC` lines, `ENDIF` and `ELSE` lines (and 6 copies of `tCancelThrown = *on;`, whose text is as before but whose enclosing block now stays open: partial). Before, `rpg_reset` did not find R in a thread-local module and the store read as an assignment, so the baseline counted them as recovered statements with a wrong text. Now R is known and the store is bookkeeping, as in every other module: the `DCL-PROC` line of a procedure that is not exported is a declaration (151 rows left the source count; the twin `FLOW/lx-subproc` shows `DCL-PROC BUMP` the same way), an `ENDIF` and an `ELSE` follow the block they close (87 rows wait for the call statements inside that stop on the status unit below), and the `DCL-PROC` line of an exported procedure is refused (126 rows, see the next section). Like for like, the baseline recovered 1,005 statements and the count is 1,490. The text of 103 newly recovered statements of asyncage was read against `source.rpgle` (conditions on pointers and based fields, assignments, `leave`, `enddo`, `return`): the operands are the ones of the source. Statements with a stop by stage: control_flow 1,434 to 753, type_inference 703 to 794, classification 47 to 217 (the status stores above), value_flow 155 and call 34 to 40. The dev emit of the six programs (`savf rpgle`): failures 535..576 to 421..450, emitted 55..62 to 170..187, one `THREAD(*CONCURRENT)` failure line in each, no clean output. The family removed: the flag word of a static symbol. The public twins turned: none (the mutation twins above only keep it from regressing). ## What stops next, and the battery 517 of the statements that still do not recover end their call with a different status unit: `99 P N 0 1; 6 1; 29 A 10 7; 37 A 15 7; 27 0; 35 R; 27 T; 37 A 10 7; 6 2`, where the plain module has `29 F 96 Q; 32 B 0 I2; 27 1; 7; 16 L; ...; 15 E` (`rpg_frames.status`). It is in no public twin, so there is no rule yet: the area A is the 429-byte static area, the field at 10 holds a five-byte text that is copied to 15 and reset, and R (the call status) is set to 0 without a test. Whether that holds for a call that can fail (`callp(e)`, `monitor`) is the question of the battery `requests/lext/thread-call` (24 cases, with plain controls, `static(*allthread)`, `thread(*serialize)`, the pointer comparison, `return *null`, `%alloc` and `dealloc`). The next rule waits for its capture. Also next: a procedure with `export` has symbol class 1, the class that `rpg_statements.recover` takes for the cycle main procedure (`is_main`). 26 of the 50 procedures of asyncage are exported. Their entry (`DCL-PROC`) and the end are read as the main's, and stop with `statement_role`. No public twin exports a procedure; `export-*` of the battery are the three cases. Gates (ship.sh): ```json { "checks": { "net_statements": 0 }, "census": { "statuses": { "recovered": 645, "partial": 0, "unresolved": 0 }, "regressions": 0, "newly_recovered": 0 }, "golden": "golden floor 173 ceiling 0 <= dump 173/5/0", "legend": "current", "emit": { "worse": [], "better": 0, "newly_clean": 0, "clean_delta": 0, "base": 588, "new": 588 }, "twins": { "groups": 826, "recovered": 804, "regressions": [], "newly": 0 }, "allgroups": { "regressions": [], "newly": [], "worse": [] }, "full_check": { "exit": 0, "seconds": 856 } } ```
A module compiled with THREAD(*CONCURRENT) has bit 0x100 on every static
data symbol. The statement readers that compare the flag word with a
constant refused the call frame, the display, monitor and file units of
such a module. Symbol.flags() reads the word without the bit; the
readers use it, the declaration emitter does not and says so.
The bit-flip tests of the display readers flip a bit that is not the thread-local one
Some checks reported errors
binjovi/ci A newer source or plan superseded this Build
291c16b45c
sean force-pushed lext from 291c16b45c
Some checks reported errors
binjovi/ci A newer source or plan superseded this Build
to f37ef72974
All checks were successful
binjovi/ci Binjovi completed the frozen plan
2026-11-10 15:49:40 +00:00
Compare
binjovi-bot manually merged commit f37ef72974 into trunk 2026-11-10 16:01:59 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sean/sourcerer!778
No description provided.