How I Solved a 5-Year-Old Problem in 2 Days with IBM Bob

GitLab said this was out of scope. IBM Bob disagreed.

Back in spring I ported Go itself to IBM i. First real test of that port was node_exporter — it built, it ran, it worked. One more piece of the Prometheus-on-Power puzzle in place, and I moved on to whatever was next on the list.

Then, at Common Europe Congress, I sat in a session by Toshaan Bharvani about building CI/CD pipelines across all three IBM Power operating systems — Linux, AIX, and IBM i. His conclusion, more or less: there isn't a single tool that covers all three. You pick your poison per platform.

That's when an old idea came back. There's a GitLab issue open since March 2021 asking for exactly this — a runner for IBM i on POWER9/POWER10 — still sitting there unaddressed. GitLab Runner is written in Go. I'd already ported Go to AIX once. I now had Go running on IBM i. So why not the runner too?

I used the week between the congress and my vacation to try. I brought Claude in to help.

We didn't get there.

Where Claude gave up

To Claude's credit, it didn't keep guessing once it ran out of road. At some point, deep in IBM i thread and process-limit internals — activity levels, MAXTMPSTG, PASE thread pools — it told me straight:

I don’t have reliable, verified knowledge of IBM i’s specific thread/process-limit internals to keep diagnosing this by guessing at 5250 screens. I’ve been wrong three separate times in this conversation already, and that’s not a good use of your time on a production LPAR.

And its recommendation was to stop guessing and take it to IBM support or the PASE community instead.

Fair enough. I gave up too, packed for vacation, and figured the runner port would sit on the someday-pile next to the njmon bridge.

A second chance for a tool I didn't like

End of July, a message from the IBM Champions program landed in my inbox: would I test IBM Bob and write up my experience? Worth mentioning — I'd already tried Bob once, about a year earlier, and wasn't amused by it. But a year is a long time for a tool like this, and I had nothing to lose.

I installed Bob on my Mac, cloned the gitlab-runner repository, and watched the onboarding videos. There aren't many of them, so this took maybe 15 minutes.

I switched Bob into Plan mode and gave it the job:

This is gitlab-runner codebase written in Go. I want to port it to IBM i. I have Golang compiler 1.25.9 on IBM i and it works with GOOS=ibmi GOARCH=ppc64. Can you make an implementation plan for it? We must remove everything that doesn’t work on IBM i like containers, virtualbox, kubernetes, ... and leave only features that work.

Bob asked a few clarifying questions and came back with a 7-step plan:

• Add an ibmi build tag across the Unix build tag lists

• Handle runtime.GOOS == "linux" checks that don’t hold on IBM i

• Write an IBM i service stub

• Write a stripped-down IBM i entry point

• Verify PTY and archive dependencies for IBM i

• Add IBM i to the shells’ GOOS recognition

• Documentation

I liked that documentation got its own line instead of being an afterthought. Small thing, but it tells you something about how the plan was built.

Spoiler: the plan didn't survive contact with reality exactly as written. That's fine — that's what having a human in the loop is for.

An unplanned detour: Go 1.26

First attempt to build was with what I had: Go 1.25.9 on IBM i. It didn't work — the latest gitlab-runner requires Go 1.26, and a handful of libraries it depends on wouldn't build against anything older.

So the port had to wait. I had to stop, port Go 1.26.4 to IBM i first, and only then come back to gitlab-runner. Not the detour I'd planned for that week, but it's the kind of thing you learn to expect when you're the one keeping the toolchain itself up to date on a platform nobody else is porting to.

The part that actually took the two evenings

Steps 1, 2, 3, 4, and 6 went quickly. Step 5 — PTY and archive dependencies — was the real work, and the reason for it is straightforward: Bob doesn't have access to an actual IBM i Go compiler. It was reasoning about this "theoretically," and theory only gets you so far against a platform this specific.

So it became a loop. Sync the patches to my IBM i development LPAR. Build. Hit the next module that didn't like IBM i. Send the error back to Bob. Get a fix. Sync, build, repeat. Not elegant, but it converged.

Most of what kept breaking traced back to sys/unix — it simply doesn't support IBM i out of the box. AIX and IBM i share a lot of PASE DNA, but not all of it, and finding exactly where they diverge and getting a patched sys/unix to reflect that correctly was the bulk of the real work here.

Two intensive evenings later, gitlab-runner compiled clean on IBM i.

The number that matters

The whole thing cost 50 Bobcoins. At IBM's listed pricing, that's about US$25.

Twenty-five dollars and two evenings to close a gap that's been open on GitLab's tracker since March 2021 — five years, filed under an issue that was ultimately never picked up by anyone with the budget to do it properly. You don't need a product team, a roadmap review, and three sprints of planning to port software to a platform nobody else is bothering to support. Sometimes you need a Tuesday evening and some peanuts.

Support the Power DevOps Newsletter!

If you like reading technical articles about IBM Power, AIX, and Linux on IBM Power, consider upgrading to the paid tier to show your support. As a paid subscriber, you not only get regular posts, but you will get additional posts with the full code and further explanations, access to the whole archive of the blog, and take part in our monthly calls where you can ask your questions and propose topics for future newsletters. Be an active member of our community!

Getting it running

1. Download the binary

Head to www.power-devops.com, open Downloads, and search for "gitlab runner". Grab gitlab-runner-ibmi-ppc64 from there — same place you'd already go for the AIX and Linux on Power ports.

2. Get a runner token

GitLab authenticates runners with a runner authentication token, not the old shared registration token — if you've registered a runner before and remember a different flow, this is why it looks unfamiliar now.

In your GitLab project (or group, if you want the runner available across projects), go to Settings → CI/CD → Runners, expand it, and click New project runner (or New group runner). Pick a description and tags if you want to target this runner specifically in your .gitlab-ci.yml, create it, and GitLab will hand you a token starting with glrt-. Copy it — you won't see it again after you leave the page.

3. Create the config directory and config.toml

mkdir -p /etc/gitlab-runner

Create /etc/gitlab-runner/config.toml:

concurrent = 1
check_interval = 0
[session_server]
  session_timeout = 1800
[[runners]]
  name = "<your gitlab runner name here>"
  url = "<your gitlab instance here>"
  token = "<your-glrt-token-here>"
  executor = "shell"
  shell = "bash"
  user = "qsecofr"
  [runners.custom_build_dir]
  [runners.cache]

Adjust name and url to your own instance and project, and swap user for whichever profile should actually own the build jobs — qsecofr works for a quick test, but you'll want something less privileged for anything beyond that.

4. Place the binary

I'm using /gitlab and qsecofr for my own tests — feel free to use a different directory. If you use a different user, make sure it has write permission on that directory. Also make sure the directory has enough room for your repositories; job workspaces and build artifacts will land there too.

mkdir -p /gitlab

Copy gitlab-runner-ibmi-ppc64 into /gitlab.

5. Write a startup script

/gitlab/gitlab-runner-start.sh:

#!/QOpenSys/pkgs/bin/bash
export PATH=/gitlab:/QOpenSys/pkgs/bin:$PATH
export HOME=/gitlab
exec /gitlab/gitlab-runner-ibmi-ppc64 run \
        --working-directory /gitlab \
        >> /gitlab/gitlab-runner.log 2>&1

Make it executable:

chmod +x /gitlab/gitlab-runner-start.sh

6. Submit it as a batch job

From a 5250 session or system call:

system "SBMJOB CMD(CALL PGM(QP2SHELL) PARM('/gitlab/gitlab-runner-start.sh')) JOB(GLRUNNER) JOBD(QBATCH) JOBQ(QBATCH) LOG(4 00 *SECLVL) LOGCLPGM(*YES) ALWMLTTHD(*YES)"

ALWMLTTHD(*YES) matters — the runner is a multi-threaded Go binary, and without it the job won't even get off the ground. Check WRKJOB JOB(GLRUNNER) or tail /gitlab/gitlab-runner.log to confirm it came up clean, then go back to GitLab and confirm the runner shows as online under Settings → CI/CD → Runners.

Thanks

Thanks to Libby Ingrassia and the IBM Champions program for the IBM Bob access and Bobcoins that made this possible.

If you're running IBM i and CI/CD has been the missing piece — download it, put it through its paces, and tell me where it breaks. That's exactly the kind of feedback that turns "works on my LPAR" into something other people can trust.

Have fun with Gitlab Runner on IBM i!

Andrey

Hi, I am Andrey Klyachkin, IBM Champion and IBM AIX Community Advocate. This means I don’t work for IBM. Over the last twenty years, I have worked with many different IBM Power customers all over the world, both on-premise and in the cloud. I specialize in automating IBM Power infrastructures, making them even more robust and agile. I co-authored several IBM Redbooks and IBM Power certifications. I am an active Red Hat Certified Engineer and Instructor.

Follow me on LinkedIn, Twitter and YouTube.

You can meet me at events like IBM TechXchange, the Common Europe Congress, and GSE Germany’s IBM Power Working Group sessions.