sourcerer

sourcerer is the IBM i source-control CLI. Point it at an IBM i and it captures the deployable text of a native application (source members, catalog metadata, SQL definitions, evidence) into a Git repository, keeps that repository in agreement with green-screen (SEU/PDM) development, and gives a CI/CD system, Binjovi here, the build, test, package and promote phases across development, staging and production libraries.

It is written for a locked-down host. The box it was proven on, pub400, denies RSTOBJ, ADDPFTRG and CRTLIB to developers and permits the compilers, SAVOBJ, CRTDUPOBJ, GENERATE_SQL and the QSYS2 catalogs. That is the profile of a production shop, so a denied operation is a recorded gap, never a failed run, and no design step depends on a restore.

It is one static Zig 0.16 binary for Linux, macOS and Windows. The IBM i transport is the zelnet package (sign-on, IFS, SQL, CL, strict source capture). Credentials come from the environment only: ZELNET_USER, ZELNET_PW and ZELNET_HOST.

Usage

sourcerer --version
sourcerer manifest validate [--manifest PATH]        # offline; default ./sourcerer.json
sourcerer connect [--manifest PATH] [--no-verify | --self-signed] [--no-tls]

manifest validate reads one file, sourcerer.json, and nothing else: no host, no clock. It prints one JSON receipt (sourcerer.manifest-validate.v1) on stdout with the counts, the build order (dependencies first) and every problem, and one summary line on stderr. Unknown and duplicate fields are parse errors, so a typo never disables a rule. Exit 0 when the manifest is valid, 2 for any problem.

$ sourcerer manifest validate --manifest sourcerer.json
sourcerer: manifest ok: rpg-lemonade-tng (15 members, 18 objects, 21 sql objects)
{"schema":"sourcerer.manifest-validate.v1","status":"done","counts":{...},"build_order":["LM_PANEL:*FILE","LM_COMMON:*MODULE",...],"problems":[]}

connect signs on to ZELNET_HOST (default pub400.com), reports the VRM and the password level, then runs the authority preflight: CHKOBJ ... OBJTYPE(*CMD) AUT(*USE) for each command a phase needs (the compilers, SAVOBJ, RSTOBJ, CRTDUPOBJ, ADDPFTRG, CRTJRN, ...) and CHKOBJ *LIB for each manifest library. A CPF9802 answer is recorded as skipped, an absent object as absent; a timeout or an unknown error fails the run. The receipt is sourcerer.connect.v1. Exit 0 done, 1 failed, 2 invalid input. --no-tls needs SOURCERER_ALLOW_PLAINTEXT=1.

The manifest describes the application once: environments and their roles, source files, members, objects with their compile commands (${LIB} is the only placeholder), SQL objects, tests and the release save file. See docs/MANIFEST.md. The verbs that follow (discover, capture, sync, build, test, verify, recover) are specified in docs/WORKFLOW.md, docs/PHASES.md and docs/RECOVERY.md and exit 2 until they exist.

Example: the lemonade stand, lost and recovered

rpg-lemonade and rpg-munchers are two synthetic IBM i applications (OPM and ILE RPG, display files, SQL tables, procedures and functions) delivered to pub400 by Binjovi. Every one of their deployments now ends with a routine source loss, so the box always looks like a shop that developed for decades and lost things: four source members disappear from the development library and two programs lose their debug data and observability (CHGPGM RMVOBS(*ALL)) in production and development.

rpg-lemonade-tng and rpg-munchers-tng are the twins recovered from that box, with no access to the original repositories. Each twin deploys through the same pipeline with the same behaviour reels, from what the box still gives up. Its recovery/README.md is a provenance table generated from receipts and hashes:

origin	what it is	rpg-lemonade-tng	rpg-munchers-tng
exact_source_captured	surviving members, exported in two strict passes and compared	15	15
generated_ddl	QSYS2.GENERATE_SQL text for tables, procedures and functions	21	12
object_metadata	catalog rows: compile attributes, bound modules, SQL precompile options	all objects	all objects
retained_binary_only	lost executables as save-file dumps with an on-box retained copy	4	4
disassembly_captured	Ghidra 11.1.2 + PowerAS bundle: instructions, layout, interpretation	LEMONADE (1,424 instructions), LM_SALEUI (1,378)	MUNCHERS (49,243), NMPERSIST (21,147)
insufficient_evidence	OPM programs (no layout adapter) and display files (no source path) stay dumps	LM_BASE, LEMSCR	NMRULE, MUNCHD

The twins compile the recovered text with commands derived from the catalog (BOUND_MODULE_INFO, PROGRAM_INFO, BOUND_SRVPGM_INFO, SYSPROGRAMSTAT) and install each lost executable from its retained copy, because the host denies RSTOBJ; the committed archive.savf is the portable form for a host that can restore.

One lost program

LM_SALEUI posts a sale. The original is 23 lines of free-format SQLRPGLE: a /copy of the parameter list, a prototype for LMSTATUS, one CALL LM_SALE(...), a COMMIT or ROLLBACK on the SQL code, and the status message. After the loss, the twin holds instead:

• recovery/objects/LM_SALEUI/archive.savf: the stripped *PGM saved from production, 130,944 bytes, sha256 recorded in capture.json;
• recovery/objects/LM_SALEUI/recovery/bundle/.../instructions.asm: 1,378 PowerPC:BE:64:AS instructions with 35 call sites, every byte proven inside the loaded image (memory_matches_image: true, complete_program_proven: false);
• .../interpretation.c: 574 lines of decompiler output, labelled as an interpretation, not source;
• the identifiers that survive in the object itself: the precompiled statement CALL LM_SALE ( : H , : H , : H , TRIM ( : H ) , : H ), COMMIT, LMSTATUS, the bound service program LM_COMMON, and the compile facts COMMIT(*CHG) OPTION(*SQL *PERIOD) ACTGRP(*CALLER).

Its surviving neighbours (LM_STK, LM_BUYUI, LM_CREDUI, ...) are byte-exact captures of the same shape, so a reconstruction kit gives a human or a model the calling convention, the copy member, the prototype and the SQL wrapper from real source; only those 23 lines have to be rewritten, and the reels prove the result. Until then the object is a reconstruction_candidate and deploys as its dump.

The recovery run itself used the pinned sean/rpg-demo toolkit and the ibmi-analyst image pins (Ghidra 11.1.2, PowerAS 0eca5b1); the recover verbs that replace it are the next slices.

The tools behind a recovery

sourcerer never runs Java. The recovery of a lost executable is a chain of pinned tools, and every link leaves a receipt with hashes:

tool	pin	role
zelnet	v0.10.18	every read and write on the box: sign-on, sql, cmd, ifs, and src export --strict, the two-pass member capture that exports a source file twice and refuses a member that changed in between
sean/rpg-demo recovery toolkit (recovery/ibmi_recover.py)	commit cd07b7f	capture-application (members, record formats, catalog rows, one QSYS2.GENERATE_SQL run per SQL object into a scratch member), capture-object (CRTSAVF, SAVOBJ ... DTACPR(*NO), CPYTOSTMF ... CVTDTA(*NONE), ifs get, catalog metadata before and after the save), recover-object, verify, lock-tools
Ghidra	11.1.2, zip sha256 219ec130…	headless analysis (analyzeHeadless) of the program image that recover-object extracts from the save file (the V7R5 uncompressed QSYS layout, 528-byte records)
PowerAS	commit 0eca5b1	the Ghidra extension for IBM i: the PowerPC:BE:64:AS processor and the loader for the serialized program object (the PGM-Kaitai structure)
exporters ExportRecovery.java, ExportInterpretation.java, VerifyPowerAS.java	in rpg-demo/recovery/ghidra/	write the loaded memory and every decoded instruction (instructions.json, instructions.asm, layout.json), the decompiler pseudocode (interpretation.c), and the processor smoke test (li, blr from eight synthetic bytes, marker POWERAS_PROCESSOR_SMOKE_PASSED)
ibmi-analyst image	registry.sean.farm/build/ibmi-analyst:v1	AlmaLinux 10, OpenJDK 21, Ghidra 11.1.2 with its Linux ARM64 natives built in the image, PowerAS, the toolkit; the gates above run at image build time

The result of recover-object is a bundle, ibmi.recovery-bundle.v1: canonical JSON, one directory per object, the manifest written last, a sha256 for every file and for the bundle. verify re-checks all of it, plus the tool lock (lock-tools records the hash of every Ghidra and JDK file) and result.json, which states what was proved:

• memory_matches_image: true: every loaded byte equals the extracted image;
• tool_files_verified: true: the tools that ran are the pinned ones;
• complete_program_proven: false, always: the listing is evidence for a human or a model, not a claim that Ghidra saw the whole program. The PowerAS loader offers a load specification for any input, so loader selection alone proves nothing; the bundle requires a complete parse, bounded segment addresses, entry points and analysis evidence.

Two limits are recorded as gaps, never hidden: an OPM program has no layout adapter (status: unsupported), and a display file has no source path from its *FILE object. Both stay save-file dumps with a retained copy.

The original source and its assembly

LM_SALEUI in rpg-lemonade, before the loss (native/LM_SQL/LM_SALEUI.sqlrpgle):

**FREE
ctl-opt option(*srcstmt:*nodebugio);
/copy LM_RPG,LM_PARMS
dcl-pr LMSTATUS char(80) extproc('LMSTATUS');
  code int(10) const;
  state char(5) const;
end-pr;
dcl-s saved int(10);
exec sql CALL ${LIB}.LM_SALE(:SALE,:DAY,:CUST,TRIM(:PROD),:QTY);
saved=SQLCOD;
STATE=SQLSTT;
MSG=LMSTATUS(saved:STATE);
if saved < 0;
  exec sql ROLLBACK;
else;
  exec sql COMMIT;
endif;
if SQLCOD < 0;
  STATE=SQLSTT;
  MSG=LMSTATUS(SQLCOD:STATE);
endif;
*inlr=*on;
return;

LM_SALEUI in rpg-lemonade-tng, after the loss (recovery/objects/LM_SALEUI/recovery/bundle/objects/SOMEARAB/PGM/LM_SALEUI/instructions.asm, 1,378 instructions; the entry and the first call site):

; IBM i recovery instruction listing
; input_sha256: 72e3e0bae69cbc503c9de36e22e40951691af0a7230423588708e80bf928d062
; language: PowerPC:BE:64:AS
; Complete program coverage has not been proved.
3885fc58f50024b0  fa41fef0  std r18, -0x110(r1)
3885fc58f50024b4  fa61fef8  std r19, -0x108(r1)
3885fc58f50024b8  fa81ff03  stmd r20, -0x40, r1
3885fc58f50024bc  7c0802a6  mfspr r0, LR
3885fc58f50024c0  f8010028  std r0, 0x28(r1)
3885fc58f50024c4  f821fea1  stdu r1, -0x160(r1)
3885fc58f50024c8  f8410020  std r2, 0x20(r1)
3885fc58f50024cc  3c005411  lis r0, 0x5411
3885fc58f50024d0  f8010008  std r0, 0x8(r1)
3885fc58f50024d4  f8210198  std r1, 0x198(r1)
3885fc58f50024d8  33ff0120  addic r31, r31, 0x120
3885fc58f50024dc  7c2001c8  txer 0x1, 0x0, 0x3
3885fc58f50024e0  41dd8183  bgtla cr7, 0xffffffffffff8180
3885fc58f50024e4  eb820030  ld r28, 0x30(r2)
3885fc58f50024e8  607d0000  ori r29, r3, 0x0
3885fc58f50024ec  e9020028  ld r8, 0x28(r2)
...
3885fc58f5002798  63070000  ori r7, r24, 0x0
3885fc58f500279c  f8d700e2  stq r6, 0xe0(r23)
3885fc58f50027a0  62e30000  ori r3, r23, 0x0
3885fc58f50027a4  4800008d  bl 0x3885fc58f5002830
3885fc58f50027a8  e9810000  ld r12, 0x0(r1)
3885fc58f50027ac  80acff8c  lwz r5, -0x74(r12)

The precompiler and the compiler leave more than machine code in the object. Converted from EBCDIC, the dump still carries the statement CALL LM_SALE ( : H , : H , : H , TRIM ( : H ) , : H ), the words COMMIT and LMSTATUS, the bound service program LM_COMMON, the source file and member names LM_SQL LM_SALEUI, and the compiler stamp 5770WDS V7R5M0. The catalog adds COMMIT(*CHG) OPTION(*SQL *PERIOD) ACTGRP(*CALLER). With the byte-exact neighbours (LM_STK is the same shape: one CALL, one status message, one COMMIT or ROLLBACK), that is enough context to rewrite the 23 lines; the reels and the compile in the release library then decide whether the rewrite is the program.

What Ghidra sees, and what it does not

Compilers on IBM i never target the hardware. OPM compilers emit an MI program template, the technology-independent Machine Interface with 16-byte tagged pointers and object semantics; ILE compilers emit W-code for the ILE optimizing translator. The Licensed Internal Code translator turns that template into the machine code of the current hardware generation and stores the result inside the object: the encapsulated program. The template, the object definition table with the names and types, and the debug views are the creation data, which IBM calls observability.

Since 1995 the translation target is PowerPC AS: PowerPC with the AS extensions, tagged 16-byte pointers guarded by a hardware tag bit, quadword pointer loads and stores, trap and tagged-load instructions. The recovered listing above is that dialect (stq, lq, ltptr, txer, stmd, lmd, td are not in stock PowerPC), the addresses are single-level-store segment addresses, and the calls to negative addresses are system entry vectors, not code in the program. That is why Ghidra needs the PowerAS processor module, and why it sees only the encapsulated layer, one level below the program's real representation. complete_program_proven: false is structural: there is no template to cross-check against.

Creation data is what makes an object portable. The 1995 CISC-to-RISC move re-translated every object from its template, and the IBM i 6.1 program conversion refused objects without creation data; those had to be recompiled. CHGPGM RMVOBS(*ALL) deletes exactly that data, which is why the routine loss uses it: every stripped object reads ALL_CREATION_DATA *UNOBS and DEBUG_DATA *NO. Two consequences:

• An observable object is a far better source than its disassembly. With DBGVIEW(*SOURCE) or *LIST the source statements sit inside the module, and the creation data carries the names. sourcerer probes those tiers before Ghidra is ever run; Ghidra is the last resort for *UNOBS objects only.
• A retained_binary_only object is not a portable binary. It runs on this hardware generation and release; the next conversion-requiring release refuses it. The only durable form is a reconstruction that passes the compile and the reels.

Build and test

zig build check          # CI parity: zig fmt --check + unit tests + CLI contracts
zig build                # zig-out/bin/sourcerer
make ci                  # the package contract + check + ReleaseSafe tests

Exit codes

0 done, 1 failed (host error, compile escape, timeout, unknown), 2 invalid input, 3 refused by a gate (drift, identity mismatch, authority denied).

Delivery

Binjovi is the CI and release provider. Open a feature pull request to trunk; the Build runs zig build check on the pinned ARM64 Zig toolchain; a Release is a Git tag on main. Nothing is merged by hand. See AGENTS.md.