The Missing Piece: Teaching Go About IBM i

Two Weeks On A Compiler. So You Don't Have To.

Two weeks ago I wrote about getting Prometheus running on AIX. Last week I wrote about the full Prometheus-based monitoring stack for IBM Power: Prometheus on Linux or AIX, node_exporter on AIX and Linux on Power, njmon bolted on for the deep stuff.

Look at that list. We have Linux on Power. We have AIX. What's missing?

IBM i.

There's no official node_exporter for IBM i. There are some community projects out there in various languages — Python, Node, the usual suspects — but no proper Prometheus-team node_exporter. That's not a coincidence. It's because there is no Google Go for IBM i. And node_exporter is written in Go.

I asked IBM about a Go port roughly two years ago. The answer was the one you've heard before if you've asked IBM for anything: "if you find a paying customer for it, we'll do it." No customers were forthcoming. The matter rested.

Until two weeks ago, when I thought: Andrey, if you want even an empty-shell node_exporter for IBM i, you need a Go compiler first. Maybe just try it.

So I tried it.

Step 0: Don't Fork. Patch.

The first decision was the most important one. There is already a thing on GitHub called ibmigo, written by Sumit Goyal, who kindly sent me the link when I started poking at this. It works by lying to the Go toolchain — it tells the compiler "I am AIX" and then patches over the places where AIX-the-real and AIX-the-PASE diverge. It's clever and it works, but it's a fork, and forks rot. Every time upstream Go ships a new release, you're rebasing a wall of changes against a moving target.

I wanted a re-baseable patch series against upstream Go, with GOOS=ibmi as a first-class build target. That means: when a new Go version comes out, I run git rebase against the new tag, fix whatever broke, ship it. No fork. No drift. Each patch a small, reviewable, single-purpose change.

This is more work upfront and a lot less work forever. So far I'm at 36 patches and counting.

Step 1: The Heap (Or, A Surprisingly Elegant Fix)

The first real problem was memory.

Go's runtime has a fast path for converting any heap pointer into a metadata-table index. It assumes heap addresses fit in 48 bits, with one fixed offset baked in per platform to handle whatever weird high-bit prefix that OS happens to use.

On Linux: zero offset.

On AIX: addresses live in the range 0x0a00000000000000 and up. So Go's runtime has a constant arenaBaseOffset = 0x0a00... for AIX, hardcoded.

I cross-compiled my first hello-world for IBM i, scp'd it to pub400, ran it, and got:

runtime: memory allocated by OS [0x0700000028000000, ...]
   not in usable address space
fatal error: memory reservation exceeds address space limit

So PASE doesn't put memory at 0x0a000... like AIX does. PASE puts it at 0x07000.... Different upper prefix, same shape of problem. The whole runtime threw up its hands on the very first allocation.

I wrote a small C probe program — mmap a few hundred pages, print the addresses — and confirmed: every PASE allocation has the prefix 0x07000000. Always. The whole 1 TiB Teraspace region lives under that prefix. PASE also ignores mmap hints, by the way, which I confirmed by hinting it five different addresses and watching it return 0x07000000... every single time, smiling pleasantly.

The fix was a one-line addition:

arenaBaseOffset = 0xffff800000000000*goarch.IsAmd64
               + 0x0a00000000000000*goos.IsAix
               + 0x0700000000000000*goos.IsIbmi   // <-- added

That's it. The blocker that has kept Go off PASE since at least 2021 — issue #45017 if you want to look it up — was a single constant. After that, hello-world ran. I almost felt cheated by how easy that was.

The universe, sensing my hubris, then arranged for everything else.

Step 2: PASE Is Mostly AIX, Until It Isn't

Phase 1 was the cross-compiler — build on my Mac, run the binary on pub400. That worked end to end with twelve patches. 8/8 of my test battery green. I felt good. I decided to do Phase 2: a native self-hosted Go toolchain, running on IBM i itself, capable of compiling itself.

That's when PASE got interesting.

A native toolchain means Go's make.bash runs on the target system. Which means every weird thing PASE does that's slightly different from AIX gets exercised, in production, all at once.

A small selection of what fell out:

uname -m returns the system serial number. Not ppc64. Not powerpc. The literal serial number of the machine — 007800001B91 on pub400. Go's build system uses uname -m to figure out what architecture it's running on. It does not know what to do with 007800001B91. Patch: hardcode gohostarch=ppc64 for ibmi, mirroring the existing AIX special case.

fstatat is a stub. Go's os package uses fstatat(dirfd, ...) to do stat-relative-to-a-directory operations. PASE's libc fstatat accepts exactly one value for dirfd: AT_FDCWD. Pass it any real directory file descriptor and it returns EINVAL. I found this out by writing — surprise — another C probe program, watching tests T1, T2, T8 pass and T3 through T7, T9, T10 fail with the same error code. The fix was to route ibmi onto Go's existing statat_other.go fallback path, the same code path used by plan9 and js/wasm — platforms that also don't have a real fstatat. The IBM i is in good company.

Errno isn't cleared on success. This one nearly cost me a full day of debugging the wrong thing. On every Unix you've ever used, a successful syscall doesn't touch errno. On PASE, a successful syscall also doesn't touch errno — but if a previous failed call left a stale value in there, that stale value is still there, and your test program will happily read it and conclude that the current call failed too. I had to add errno = 0 before every probe call. Lesson learned, applied to every probe written since.

The XCOFF loader races itself. This is the one I'm currently sitting on. PASE's dynamic loader is not thread-safe when multiple processes simultaneously resolve imports from the same shared library. Go's build system likes to do exactly that — it spawns parallel compile jobs, each of which forks and execs a fresh compile binary, all of which need libpthreads.a resolved at the same moment. About 50% of the time, two of them step on each other and the build dies with a loader error code. Workaround: pin cmd/go to single-threaded build (-p 1) on ibmi. It's slow but it's deterministic. A real fix is going to need either retry logic in the runtime's exec path or a pre-warm of the loader's cache.

There are more patches like this. I'll spare you them. The shape is always the same: write a probe, confirm what PASE actually does, write the smallest possible patch, document, move on.

Step 3: The Reference Port That Already Did Half The Work

About halfway through Phase 2 I went back to look at ibmigo's git history more carefully — Sumit’s port, the existing fork I'd decided not to use as my base — and found that he had independently discovered most of the same PASE quirks I was hitting. Same -p=1 workaround for the loader race. Same maxbg=1 for the GC. Even the same trick of retrying mkdir in a loop because PASE's syscalls are sometimes spuriously racy.

I'd been worried I was the only one seeing some of this. Turns out: no, this is just what PASE is. Sumit and I have shared confused experiences across years of disconnected effort, and we both ended up at almost identical workarounds. There is some comfort in that.

Common Europe Congress 2026 is there!

The agenda is published! Do you want to know where AIX is going to? It means you MUST visit the Common Europe Congress in Lyon, France. There will be sessions about new AIX features and open source community development. We will talk about AIX and IBM Power automation and Zero Downtime for AIX. Join me in Lyon!

Where We Are, And Where We're Going

The current state: cross-compile from my Mac to IBM i works cleanly. A native Go toolchain on IBM i builds, but only with GOMAXPROCS=1 and -p=1, and make.bash succeeds maybe half the time because of the loader race. I don't yet trust it enough to publish it. Alpha would be a generous label.

But it compiles things. It runs things. It is real Go, talking to real IBM i.

Once I'm confident the loader race is properly worked around — not just papered over — I'll publish the port. Anyone who wants to download the port and see what happens will be able to do so. After that comes node_exporter. And after that, the IBM i finally gets to join its AIX and Linux siblings in the same Prometheus dashboard, scraped on the same port 9100, looking like every other host on the network.

Wait a bit. You'll most probably get it.

Thanks

This work would have been considerably harder, and possibly impossible at this speed, without pub400.com — the free public IBM i system run by Holger Scherer. Every probe program in this article ran on pub400. Every "okay but does PASE actually do this?" question got answered there. Holger has been running this service for years, free of charge, for the IBM i community. If you're doing anything with IBM i and you don't have your own LPAR handy, pub400 is where you go. Holger, thank you.

Thanks also to Sumit Goyal, who built the ibmigo port and sent me the link when I started this. Several of the trickier PASE workarounds in my patch series — the loader-race mitigations especially — exist in something like their current form because Sumit got there first and his repo is public. Independent confirmation that you're not hallucinating a problem is worth a lot when you're three days into debugging a 50%-flaky build.

Have fun porting software to your favorite platform!

Andrey

Hi, I am Andrey Klyachkin, IBM Champion and IBM AIX Community Advocate. This means I don’t work for IBM. Over the last twenty years, I have worked with many different IBM Power customers all over the world, both on-premise and in the cloud. I specialize in automating IBM Power infrastructures, making them even more robust and agile. I co-authored several IBM Redbooks and IBM Power certifications. I am an active Red Hat Certified Engineer and Instructor.

Follow me on LinkedIn, Twitter and YouTube.

You can meet me at events like IBM TechXchange, the Common Europe Congress, and GSE Germany’s IBM Power Working Group sessions.