- Zig 95.6%
- Shell 3.3%
- C++ 0.5%
- Objective-C 0.4%
- C 0.1%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| ci | ||
| docs | ||
| gui | ||
| macos | ||
| packaging | ||
| src | ||
| tests/cli | ||
| tools | ||
| windows | ||
| .gitattributes | ||
| .gitignore | ||
| AGENTS.md | ||
| build.zig | ||
| build.zig.zon | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| LICENSE | ||
| Makefile | ||
| README.md | ||
| STYLE.md | ||
| VOICE.md | ||
Sourcerer
Sourcerer exports an IBM i application into a directory of ordinary files on Windows, macOS, or Linux, so the application can live in source control and move through the same delivery pipeline as everything else you build.
The directory it writes is meant to be committed. Git is the canonical example in this README, but the output is just files: Subversion, Perforce, or anything else works the same way. From there, Jenkins, ARCAD, GitLab, Azure DevOps, or whatever runs your pipeline can compile the members that have source, restore the objects that do not, and diff every change ever made.
For each object, Sourcerer writes the best evidence that still exists:
- Source members come out as text files, converted on the host to UTF-8 with LF line endings.
- Programs that kept a listing debug view (
DBGVIEW(*LIST)or(*ALL)) get their complete source member back, with its sequence numbers and change dates. - Programs whose source and observability are gone get what the compiled object still proves: recovered statements, the MI program of an OPM object, literals, imports, procedure boundaries, and optionally a disassembly. See Recovering stripped programs.
- Every object gets
object.savf(in a host dump) and a receipt,object.json, that lists each file written with its hash and the result of each step.sourcerer checkre-verifies the whole tree against those receipts at any time.
Sourcerer reads the application and never writes source or programs back to the host. It makes no commits and pushes nothing; your pipeline owns that. Dumps contain your business logic and string literals. Keep them in a private repository.
Install
Download from pkg.sean.farm/sourcerer. Each release has a Windows x86_64 executable, macOS installer packages for Apple silicon and Intel, and checksums. On Linux, build from source.
On Windows, rename the download and put it on your PATH:
Rename-Item sourcerer-0.1.244-windows-x86_64.exe sourcerer.exe
sourcerer --version
One honest caveat: the Windows executable is cross-compiled and the test suite runs on Linux and macOS hosts, not on Windows. The examples below are the supported commands; if one misbehaves on Windows, that is a bug worth reporting.
Dump a library from a host
A host dump signs on with the IBM i host servers and needs an account with authority to read and save the selected objects. Credentials come from the environment only; they are never command-line arguments.
In PowerShell 7:
$env:ZELNET_HOST = 'ibmi.example.net'
$env:ZELNET_USER = 'MYUSER'
$env:ZELNET_PW = Read-Host -MaskInput 'IBM i password'
(In Windows PowerShell 5.1, which has no -MaskInput:
$pw = Read-Host -AsSecureString 'IBM i password'; $env:ZELNET_PW = [System.Net.NetworkCredential]::new('', $pw).Password.)
TLS with certificate verification is on by default. For a host with a
self-signed certificate, add --self-signed. The
dump reference has the full connection and
trust options.
Start with one object. Library and object names are the uppercase IBM i
names; applib is refused, APPLIB is right:
sourcerer dump APPLIB START.PGM -o applib
sourcerer check applib
Then take the whole library:
sourcerer dump APPLIB -o applib
sourcerer check applib
A whole-library dump also removes the directories of objects that no
longer exist, so the output tracks the library. A dump with explicit
object names merges those objects in and removes nothing. Supported object
types are PGM, SRVPGM, MODULE, and FILE. Database files get their
metadata and DDL, not their rows: this is an application export, not a
database backup.
The dump creates private temporary objects on the host for the export (a save file, an outfile, a stream file) and proves their removal before it finishes. A cleanup that cannot be confirmed fails the dump.
When you are done:
Remove-Item Env:ZELNET_PW
Dump save files without a host
If what you have is save files — from SAVOBJ ... DTACPR(*NO), copied to
the IFS with CPYTOSTMF ... CVTDTA(*NONE) and downloaded — Sourcerer
reads them directly, no host and no credentials:
sourcerer dump --savf archive.savf -o out/archive
sourcerer check out/archive
--savf takes exactly one save file. For a directory full of them, give
each save file its own output directory:
Get-ChildItem *.savf | ForEach-Object {
sourcerer dump --savf $_.FullName -o "out/$($_.BaseName)"
sourcerer check "out/$($_.BaseName)"
}
Do not point two save files of the same library at one output directory:
the receipts track one source archive per dump, and check will refuse
the first file's objects after the second dump.
An offline dump reports each object as partial. That is expected: the
host catalogue and reference queries cannot run against a file, so the
metadata and references steps are skipped. A partial offline dump is
still exit 0.
Add --decomp to either kind of dump for a disassembly and a C-like
interpretation of each program, written under decomp/.
Text, encodings, and filenames
The short version: everything Sourcerer writes is UTF-8 with LF line endings, and the output tree is safe on every filesystem Git runs on. The details, because this is where IBM i meets three decades of PC decisions:
- EBCDIC to UTF-8. In a host dump, conversion happens on the IBM i
itself: members and outfiles come down through
CPYTOSTMFandCPYTOIMPFwithSTMFCCSID(1208)and LF record delimiters, and DDL throughGENERATE_SQLwithCCSID 1208. In an offline dump there is no host to convert, so text inside object images is decoded as CCSID 37. - Line endings. Every text file is written with LF, on Windows too. There is no option for CRLF, on purpose: the same library dumped from Windows and from Linux produces byte-identical files.
- Filename case. Object directories are the uppercase IBM i names
(
START.PGM); the files inside them are fixed lowercase names (object.json,statements.rpgle); member extensions are lowercase. No two paths in a dump ever differ only by case, so NTFS and the default case-insensitive APFS hold the tree exactly as a case-sensitive Linux filesystem does. - One real Windows limitation.
CON,PRN,AUX,NUL,COM1throughCOM9, andLPT1throughLPT9are reserved device names on Windows and perfectly legal IBM i object names. Sourcerer refuses to write such a path on Windows and the dump stops with exit 1. Dump a library containing one of those objects from macOS or Linux.
Git on Windows
Git for Windows defaults to converting line endings, and that default
breaks this workflow: sourcerer check re-hashes every file against its
receipt, so a checkout that rewrites LF to CRLF fails check on every
text file with exit 3.
Two rules make Windows, macOS, and Linux clones of the same dump repository byte-identical:
-
Turn off line-ending conversion for the repository (or globally, or at clone time with
git clone -c core.autocrlf=false ...):git config core.autocrlf false -
Pin it in the repository so no collaborator's global config can undo it, and keep the dump in a subdirectory.
checktreats any file in the dump directory that no receipt lists as a finding (only.gitis ignored), so.gitattributesbelongs at the repository root, not in the dump:repo/ ├── .gitattributes # contains: * -text ├── README.md └── applib/ # sourcerer dump APPLIB -o applib* -texttells Git to never convert anything, which is exactly right for a tree that mixes text receipts with binaryobject.savffiles.
With that in place, sourcerer check applib passes on a fresh clone on
any platform, which makes it a cheap CI job: dump on a schedule, check
on every pull request.
Recovering stripped programs
Running programs whose source members are gone, and whose observability
was removed with CHGPGM RMVOBS(*ALL), are the worst case this tool is
built for — and the reason it exists.
The short version of what makes recovery possible: RMVOBS(*ALL) does
not delete as much as the documentation says. On current systems the
compiler's creation data survives inside the object, compressed with an
undocumented LZ77-family scheme that Sourcerer decodes. For an OPM
program that means the complete MI program with its field and label
names. For an ILE RPG program it means every statement's number, its
operations, its operand types, and its literals — though not its variable
names. How this was discovered, reverse-engineered, and tested is written
up in the recovery story. How each statement is proved, and what a partial status and its failure
records mean, is in the proof engine.
You can see it on a fixture in this repository, no IBM i required. The
fixture is a small RPG program saved after RMVOBS(*ALL):
git clone https://code.sean.farm/sean/sourcerer.git
cd sourcerer
sourcerer dump --savf src/testdata/savf/SC-ILE/TSTRIP-STRIPPED/archive.savf -o out/tstrip
sourcerer check out/tstrip
Get-Content out/tstrip/TSTRIPS.PGM/statements.rpgle -TotalCount 14
The statements file begins (this is the committed golden output, src/testdata/savf/SC-ILE/TSTRIP-STRIPPED/statements.rpgle, which the test suite asserts):
// TSTRIPS.PGM: recovered statements; not compilable source.
// Source statements: recovered 36; partial 2; unresolved 0.
// Separate numbered groups: declarations 2; compiler-generated 2.
// S = static, A = automatic, P = parameter, T = other; numbers identify symbols.
// .N is byte N of a data structure. Original variable names are not recovered.
// Each evidence reference names a complete group in statements.raw.txt.
// Module 1, procedure 1: SCORE
// declaration; recovered; evidence G2
5600 DCL-PROC SCORE
// source; recovered; evidence G3
6100 IF P261 > 2
// source; recovered; evidence G4
6200 A260 = P261 * 7.25
The numbers in the left column are the original source statement numbers.
P261 is parameter number 261, not a guess at a name. Each statement
cites its evidence: a group, a contiguous range of creation-data
operations, printed in full in statements.raw.txt so you can inspect
the basis for every line. G1 here is the unnumbered procedure prologue,
which is why the first readable statement cites G2.
Every statement carries one of three statuses. recovered means every operation in the group is explained by a proven rule. partial means some are. unresolved means none are — which is work to investigate in the raw evidence, not a blank to fill with something plausible.
sourcerer savf rpgle FILE OBJ.PGM -o recovered.rpgle goes one step
further and emits fully free-form RPG IV from the same evidence, with
explicit failure lines where the evidence runs out. It is a draft for a
programmer to finish against the raw evidence, not the original source.
Know the limits before you plan a project around this:
- Variable names are never recovered for a stripped ILE program. The
symbol identifiers (
P261,A260) are stable, but naming things again is the human's job. (A stripped OPM program is the happy exception: its MI template keeps real field and label names.) - Some facts are provably absent. Two programs that differ only in a packed field's decimal positions can compile to byte-identical creation data, so no tool can recover that distinction; statement numbers of a stripped OPM program are deleted outright. The recovery story lists what is measured as gone.
- Some forms are refused rather than guessed.
OPTIMIZE(*FULL)objects, for one. Exit 3 with a reason beats silently wrong output. checkproves integrity, not correctness. It verifies that the files match their receipts. A perfectly preserved wrong answer is still a wrong answer: test any rewritten program against the behavior of the running original.- This is active work. Recovery rules improve release to release, and the counts above are for this fixture, not a rate to expect on your application. At v0.1.409 the public matrix has 449 rows, 271 of them matching (the proof engine, section 4). The recovery story ends with the current open problems and will be updated as they close.
A window, on Windows and macOS
Sourcerer has a native window for the same work, on Windows and on
macOS: connect to a
host, search and pick a library, check the objects to dump, choose an
output directory, dump with a progress bar, stop, verify the receipts,
and read what evidence each object got. It drives the same dump engine
and writes the same receipts as the commands above; Verify receipts
is sourcerer check.
With no host at all, drop .savf files on the window (or use Open save
file…): each is dumped into the output folder, as dump --savf does.
Double-click an object to see its evidence side by side, such as
statements.rpgle beside statements.raw.txt. Recover source writes
the RPG recovered from the selected program, with the DDS rebuilt for its
files, into a folder beside the dump, <output>-recovered, as
savf rpgle does: a draft for a person to finish, never inside the dump
tree, so check stays clean.
The Windows executable and the macOS packages hold the window, and the
same sourcerer executable is both tools: run it from PowerShell or a
shell and it is the command line above; double-click sourcerer.exe,
open sourcerer.app (built below), or run sourcerer --gui, and the window
opens. A script or a pipe gets the usage, never the window. A
double-clicked sourcerer.exe hides the console window Windows gives it.
The credentials you type go to the host session only, never to a file, a
log or a receipt.
To build the window from source on a Mac (Zig 0.17 and the pinned Zelnet checkout, as in Building from source):
zig build -Dgui macos-run
That assembles zig-out/sourcerer.app and opens it. The Windows window
cross-compiles from any host, with no Windows SDK:
zig build -Dgui -Dtarget=x86_64-windows
Exit codes
For pipelines: 0 done, 1 operation failed, 2 invalid input or
missing environment, 3 refused by a safety gate (including check
findings). A partial offline dump is 0; missing required evidence is
never a silent success.
Building from source
Use Zig 0.17. From the repository root, put the pinned Zelnet source in
zelnet/. The pin is in build.zig; the local dependency path is in
build.zig.zon. Access to both repositories is required.
zelnet_version=$(sed -n 's/^const zelnet_pin = "\(.*\)";/\1/p' build.zig)
git clone --depth 1 --branch "v${zelnet_version}" \
https://code.sean.farm/sean/zelnet.git zelnet
zig build
zig-out/bin/sourcerer --help
make ci
make ci runs the package contract, formatting and CLI checks, and the
debug and safe-mode tests. See STYLE.md before changing code.
Further reading
- Dump reference: every command, output file, receipt field, host operation, and exit code.
- The recovery story: how statement recovery from stripped programs was discovered, reverse-engineered, and tested.
- The compressed template: the format of the compressed creation-data record, and how to measure it yourself.
- Comparison reference: what
sourcerer compareproves about two saved programs, and what it refuses. - License, decompiler notices, and instruction-decoder notices.