IBM i source control CLI (Zig): member export and import over the zelnet Host Server client
  • Zig 95.6%
  • Shell 3.3%
  • C++ 0.5%
  • Objective-C 0.4%
  • C 0.1%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Sean OMeara 760f7aa1e9
All checks were successful
binjovi/ci Binjovi completed the frozen plan
Regenerate the operations legend
2026-11-20 20:04:59 +01:00
ci Guard the source and the output against the upstream project name 2026-10-14 19:23:30 +02:00
docs Regenerate the operations legend 2026-11-20 20:04:59 +01:00
gui fix(gui): a bugfix round from an independent review of the windows 2026-10-21 00:50:02 +02:00
macos fix(gui): a bugfix round from an independent review of the windows 2026-10-21 00:50:02 +02:00
packaging Remove the packaging code of the Java runtime 2026-10-13 21:52:31 +02:00
src Keep optional declarations within source layout 2026-11-20 19:12:55 +01:00
tests/cli tools/recompile-compare.sh: RECOMPILE_LISTING_LINES sets the listing lines printed for a failed compile 2026-11-11 09:21:16 +01:00
tools tools/recompile-compare.sh: RECOMPILE_LISTING_LINES sets the listing lines printed for a failed compile 2026-11-11 09:21:16 +01:00
windows fix(gui): a bugfix round from an independent review of the windows 2026-10-21 00:50:02 +02:00
.gitattributes Merge src/root.zig with the union driver 2026-11-13 13:22:38 +01:00
.gitignore Package native macOS analysis runtime 2026-09-26 13:31:51 +02:00
AGENTS.md Restore the v0.1.155 Sourcerer README 2026-10-17 19:22:15 +02:00
build.zig Recover retained PCML parameter names and compare the module interface 2026-11-19 07:51:37 +01:00
build.zig.zon chore: migrate Sourcerer to Zig 0.17.0 2026-10-17 06:34:38 +02:00
CHANGELOG.md Binjovi queue 3e32ce8a900855e2: merge pull request #576 2026-10-20 19:23:42 +00:00
CLAUDE.md feat: the sourcerer skeleton on the zelnet package 2026-09-14 06:48:50 +02:00
LICENSE Package native macOS analysis runtime 2026-09-26 13:31:51 +02:00
Makefile chore: migrate Sourcerer to Zig 0.17.0 2026-10-17 06:34:38 +02:00
README.md Document the proof engine and refresh the recovery status 2026-11-03 13:47:10 +01:00
STYLE.md test: make allocation failure sweeps independent of resize behavior 2026-10-18 20:26:13 +02:00
VOICE.md Rewrite the README in Sean’s teaching voice 2026-10-15 12:48:04 +02:00

Sourcerer

Sourcerer exports an IBM i application into a directory of ordinary files on Windows, macOS, or Linux, so the application can live in source control and move through the same delivery pipeline as everything else you build.

The directory it writes is meant to be committed. Git is the canonical example in this README, but the output is just files: Subversion, Perforce, or anything else works the same way. From there, Jenkins, ARCAD, GitLab, Azure DevOps, or whatever runs your pipeline can compile the members that have source, restore the objects that do not, and diff every change ever made.

For each object, Sourcerer writes the best evidence that still exists:

  • Source members come out as text files, converted on the host to UTF-8 with LF line endings.
  • Programs that kept a listing debug view (DBGVIEW(*LIST) or (*ALL)) get their complete source member back, with its sequence numbers and change dates.
  • Programs whose source and observability are gone get what the compiled object still proves: recovered statements, the MI program of an OPM object, literals, imports, procedure boundaries, and optionally a disassembly. See Recovering stripped programs.
  • Every object gets object.savf (in a host dump) and a receipt, object.json, that lists each file written with its hash and the result of each step. sourcerer check re-verifies the whole tree against those receipts at any time.

Sourcerer reads the application and never writes source or programs back to the host. It makes no commits and pushes nothing; your pipeline owns that. Dumps contain your business logic and string literals. Keep them in a private repository.

Install

Download from pkg.sean.farm/sourcerer. Each release has a Windows x86_64 executable, macOS installer packages for Apple silicon and Intel, and checksums. On Linux, build from source.

On Windows, rename the download and put it on your PATH:

Rename-Item sourcerer-0.1.244-windows-x86_64.exe sourcerer.exe
sourcerer --version

One honest caveat: the Windows executable is cross-compiled and the test suite runs on Linux and macOS hosts, not on Windows. The examples below are the supported commands; if one misbehaves on Windows, that is a bug worth reporting.

Dump a library from a host

A host dump signs on with the IBM i host servers and needs an account with authority to read and save the selected objects. Credentials come from the environment only; they are never command-line arguments.

In PowerShell 7:

$env:ZELNET_HOST = 'ibmi.example.net'
$env:ZELNET_USER = 'MYUSER'
$env:ZELNET_PW   = Read-Host -MaskInput 'IBM i password'

(In Windows PowerShell 5.1, which has no -MaskInput: $pw = Read-Host -AsSecureString 'IBM i password'; $env:ZELNET_PW = [System.Net.NetworkCredential]::new('', $pw).Password.)

TLS with certificate verification is on by default. For a host with a self-signed certificate, add --self-signed. The dump reference has the full connection and trust options.

Start with one object. Library and object names are the uppercase IBM i names; applib is refused, APPLIB is right:

sourcerer dump APPLIB START.PGM -o applib
sourcerer check applib

Then take the whole library:

sourcerer dump APPLIB -o applib
sourcerer check applib

A whole-library dump also removes the directories of objects that no longer exist, so the output tracks the library. A dump with explicit object names merges those objects in and removes nothing. Supported object types are PGM, SRVPGM, MODULE, and FILE. Database files get their metadata and DDL, not their rows: this is an application export, not a database backup.

The dump creates private temporary objects on the host for the export (a save file, an outfile, a stream file) and proves their removal before it finishes. A cleanup that cannot be confirmed fails the dump.

When you are done:

Remove-Item Env:ZELNET_PW

Dump save files without a host

If what you have is save files — from SAVOBJ ... DTACPR(*NO), copied to the IFS with CPYTOSTMF ... CVTDTA(*NONE) and downloaded — Sourcerer reads them directly, no host and no credentials:

sourcerer dump --savf archive.savf -o out/archive
sourcerer check out/archive

--savf takes exactly one save file. For a directory full of them, give each save file its own output directory:

Get-ChildItem *.savf | ForEach-Object {
    sourcerer dump --savf $_.FullName -o "out/$($_.BaseName)"
    sourcerer check "out/$($_.BaseName)"
}

Do not point two save files of the same library at one output directory: the receipts track one source archive per dump, and check will refuse the first file's objects after the second dump.

An offline dump reports each object as partial. That is expected: the host catalogue and reference queries cannot run against a file, so the metadata and references steps are skipped. A partial offline dump is still exit 0.

Add --decomp to either kind of dump for a disassembly and a C-like interpretation of each program, written under decomp/.

Text, encodings, and filenames

The short version: everything Sourcerer writes is UTF-8 with LF line endings, and the output tree is safe on every filesystem Git runs on. The details, because this is where IBM i meets three decades of PC decisions:

  • EBCDIC to UTF-8. In a host dump, conversion happens on the IBM i itself: members and outfiles come down through CPYTOSTMF and CPYTOIMPF with STMFCCSID(1208) and LF record delimiters, and DDL through GENERATE_SQL with CCSID 1208. In an offline dump there is no host to convert, so text inside object images is decoded as CCSID 37.
  • Line endings. Every text file is written with LF, on Windows too. There is no option for CRLF, on purpose: the same library dumped from Windows and from Linux produces byte-identical files.
  • Filename case. Object directories are the uppercase IBM i names (START.PGM); the files inside them are fixed lowercase names (object.json, statements.rpgle); member extensions are lowercase. No two paths in a dump ever differ only by case, so NTFS and the default case-insensitive APFS hold the tree exactly as a case-sensitive Linux filesystem does.
  • One real Windows limitation. CON, PRN, AUX, NUL, COM1 through COM9, and LPT1 through LPT9 are reserved device names on Windows and perfectly legal IBM i object names. Sourcerer refuses to write such a path on Windows and the dump stops with exit 1. Dump a library containing one of those objects from macOS or Linux.

Git on Windows

Git for Windows defaults to converting line endings, and that default breaks this workflow: sourcerer check re-hashes every file against its receipt, so a checkout that rewrites LF to CRLF fails check on every text file with exit 3.

Two rules make Windows, macOS, and Linux clones of the same dump repository byte-identical:

  1. Turn off line-ending conversion for the repository (or globally, or at clone time with git clone -c core.autocrlf=false ...):

    git config core.autocrlf false
    
  2. Pin it in the repository so no collaborator's global config can undo it, and keep the dump in a subdirectory. check treats any file in the dump directory that no receipt lists as a finding (only .git is ignored), so .gitattributes belongs at the repository root, not in the dump:

    repo/
    ├── .gitattributes      # contains: * -text
    ├── README.md
    └── applib/             # sourcerer dump APPLIB -o applib
    

    * -text tells Git to never convert anything, which is exactly right for a tree that mixes text receipts with binary object.savf files.

With that in place, sourcerer check applib passes on a fresh clone on any platform, which makes it a cheap CI job: dump on a schedule, check on every pull request.

Recovering stripped programs

Running programs whose source members are gone, and whose observability was removed with CHGPGM RMVOBS(*ALL), are the worst case this tool is built for — and the reason it exists.

The short version of what makes recovery possible: RMVOBS(*ALL) does not delete as much as the documentation says. On current systems the compiler's creation data survives inside the object, compressed with an undocumented LZ77-family scheme that Sourcerer decodes. For an OPM program that means the complete MI program with its field and label names. For an ILE RPG program it means every statement's number, its operations, its operand types, and its literals — though not its variable names. How this was discovered, reverse-engineered, and tested is written up in the recovery story. How each statement is proved, and what a partial status and its failure records mean, is in the proof engine.

You can see it on a fixture in this repository, no IBM i required. The fixture is a small RPG program saved after RMVOBS(*ALL):

git clone https://code.sean.farm/sean/sourcerer.git
cd sourcerer
sourcerer dump --savf src/testdata/savf/SC-ILE/TSTRIP-STRIPPED/archive.savf -o out/tstrip
sourcerer check out/tstrip
Get-Content out/tstrip/TSTRIPS.PGM/statements.rpgle -TotalCount 14

The statements file begins (this is the committed golden output, src/testdata/savf/SC-ILE/TSTRIP-STRIPPED/statements.rpgle, which the test suite asserts):

// TSTRIPS.PGM: recovered statements; not compilable source.
// Source statements: recovered 36; partial 2; unresolved 0.
// Separate numbered groups: declarations 2; compiler-generated 2.
// S = static, A = automatic, P = parameter, T = other; numbers identify symbols.
// .N is byte N of a data structure. Original variable names are not recovered.
// Each evidence reference names a complete group in statements.raw.txt.

// Module 1, procedure 1: SCORE
// declaration; recovered; evidence G2
      5600  DCL-PROC SCORE
// source; recovered; evidence G3
      6100  IF P261 > 2
// source; recovered; evidence G4
      6200    A260 = P261 * 7.25

The numbers in the left column are the original source statement numbers. P261 is parameter number 261, not a guess at a name. Each statement cites its evidence: a group, a contiguous range of creation-data operations, printed in full in statements.raw.txt so you can inspect the basis for every line. G1 here is the unnumbered procedure prologue, which is why the first readable statement cites G2.

Every statement carries one of three statuses. recovered means every operation in the group is explained by a proven rule. partial means some are. unresolved means none are — which is work to investigate in the raw evidence, not a blank to fill with something plausible.

sourcerer savf rpgle FILE OBJ.PGM -o recovered.rpgle goes one step further and emits fully free-form RPG IV from the same evidence, with explicit failure lines where the evidence runs out. It is a draft for a programmer to finish against the raw evidence, not the original source.

Know the limits before you plan a project around this:

  • Variable names are never recovered for a stripped ILE program. The symbol identifiers (P261, A260) are stable, but naming things again is the human's job. (A stripped OPM program is the happy exception: its MI template keeps real field and label names.)
  • Some facts are provably absent. Two programs that differ only in a packed field's decimal positions can compile to byte-identical creation data, so no tool can recover that distinction; statement numbers of a stripped OPM program are deleted outright. The recovery story lists what is measured as gone.
  • Some forms are refused rather than guessed. OPTIMIZE(*FULL) objects, for one. Exit 3 with a reason beats silently wrong output.
  • check proves integrity, not correctness. It verifies that the files match their receipts. A perfectly preserved wrong answer is still a wrong answer: test any rewritten program against the behavior of the running original.
  • This is active work. Recovery rules improve release to release, and the counts above are for this fixture, not a rate to expect on your application. At v0.1.409 the public matrix has 449 rows, 271 of them matching (the proof engine, section 4). The recovery story ends with the current open problems and will be updated as they close.

A window, on Windows and macOS

Sourcerer has a native window for the same work, on Windows and on macOS: connect to a host, search and pick a library, check the objects to dump, choose an output directory, dump with a progress bar, stop, verify the receipts, and read what evidence each object got. It drives the same dump engine and writes the same receipts as the commands above; Verify receipts is sourcerer check.

With no host at all, drop .savf files on the window (or use Open save file…): each is dumped into the output folder, as dump --savf does. Double-click an object to see its evidence side by side, such as statements.rpgle beside statements.raw.txt. Recover source writes the RPG recovered from the selected program, with the DDS rebuilt for its files, into a folder beside the dump, <output>-recovered, as savf rpgle does: a draft for a person to finish, never inside the dump tree, so check stays clean.

The Windows executable and the macOS packages hold the window, and the same sourcerer executable is both tools: run it from PowerShell or a shell and it is the command line above; double-click sourcerer.exe, open sourcerer.app (built below), or run sourcerer --gui, and the window opens. A script or a pipe gets the usage, never the window. A double-clicked sourcerer.exe hides the console window Windows gives it. The credentials you type go to the host session only, never to a file, a log or a receipt.

To build the window from source on a Mac (Zig 0.17 and the pinned Zelnet checkout, as in Building from source):

zig build -Dgui macos-run

That assembles zig-out/sourcerer.app and opens it. The Windows window cross-compiles from any host, with no Windows SDK:

zig build -Dgui -Dtarget=x86_64-windows

Exit codes

For pipelines: 0 done, 1 operation failed, 2 invalid input or missing environment, 3 refused by a safety gate (including check findings). A partial offline dump is 0; missing required evidence is never a silent success.

Building from source

Use Zig 0.17. From the repository root, put the pinned Zelnet source in zelnet/. The pin is in build.zig; the local dependency path is in build.zig.zon. Access to both repositories is required.

zelnet_version=$(sed -n 's/^const zelnet_pin = "\(.*\)";/\1/p' build.zig)
git clone --depth 1 --branch "v${zelnet_version}" \
  https://code.sean.farm/sean/zelnet.git zelnet
zig build
zig-out/bin/sourcerer --help
make ci

make ci runs the package contract, formatting and CLI checks, and the debug and safe-mode tests. See STYLE.md before changing code.

Further reading