Look at that list. We have Linux on Power. We have AIX. What's missing?

IBM i.

There's no official node_exporter for IBM i. There are some community projects out there in various languages — Python, Node, the usual suspects — but no proper Prometheus-team node_exporter. That's not a coincidence. It's because there is no Google Go for IBM i. And node_exporter is written in Go.

I asked IBM about a Go port roughly two years ago. The answer was the one you've heard before if you've asked IBM for anything: "if you find a paying customer for it, we'll do it." No customers were forthcoming. The matter rested.

Until two weeks ago, when I thought: Andrey, if you want even an empty-shell node_exporter for IBM i, you need a Go compiler first. Maybe just try it.

So I tried it.

Step 0: Don't Fork. Patch.

The first decision was the most important one. There is already a thing on GitHub called ibmigo, written by Sumit Goyal, who kindly sent me the link when I started poking at this. It works by lying to the Go toolchain — it tells the compiler "I am AIX" and then patches over the places where AIX-the-real and AIX-the-PASE diverge. It's clever and it works, but it's a fork, and forks rot. Every time upstream Go ships a new release, you're rebasing a wall of changes against a moving target.

I wanted a re-baseable patch series against upstream Go, with GOOS=ibmi as a first-class build target. That means: when a new Go version comes out, I run git rebase against the new tag, fix whatever broke, ship it. No fork. No drift. Each patch a small, reviewable, single-purpose change.

This is more work upfront and a lot less work forever. So far I'm at 36 patches and counting.

Step 1: The Heap (Or, A Surprisingly Elegant Fix)

The first real problem was memory.

Go's runtime has a fast path for converting any heap pointer into a metadata-table index. It assumes heap addresses fit in 48 bits, with one fixed offset baked in per platform to handle whatever weird high-bit prefix that OS happens to use.

On Linux: zero offset.

On AIX: addresses live in the range 0x0a00000000000000 and up. So Go's runtime has a constant arenaBaseOffset = 0x0a00... for AIX, hardcoded.

I cross-compiled my first hello-world for IBM i, scp'd it to pub400, ran it, and got:

runtime: memory allocated by OS [0x0700000028000000, ...]
   not in usable address space
fatal error: memory reservation exceeds address space limit

So PASE doesn't put memory at 0x0a000... like AIX does. PASE puts it at 0x07000.... Different upper prefix, same shape of problem. The whole runtime threw up its hands on the very first allocation.

I wrote a small C probe program — mmap a few hundred pages, print the addresses — and confirmed: every PASE allocation has the prefix 0x07000000. Always. The whole 1 TiB Teraspace region lives under that prefix. PASE also ignores mmap hints, by the way, which I confirmed by hinting it five different addresses and watching it return 0x07000000... every single time, smiling pleasantly.

The fix was a one-line addition:

arenaBaseOffset = 0xffff800000000000*goarch.IsAmd64
               + 0x0a00000000000000*goos.IsAix
               + 0x0700000000000000*goos.IsIbmi   // <-- added

That's it. The blocker that has kept Go off PASE since at least 2021 — issue #45017 if you want to look it up — was a single constant. After that, hello-world ran. I almost felt cheated by how easy that was.

The universe, sensing my hubris, then arranged for everything else.

Step 2: PASE Is Mostly AIX, Until It Isn't

Phase 1 was the cross-compiler — build on my Mac, run the binary on pub400. That worked end to end with twelve patches. 8/8 of my test battery green. I felt good. I decided to do Phase 2: a native self-hosted Go toolchain, running on IBM i itself, capable of compiling itself.

That's when PASE got interesting.

A native toolchain means Go's make.bash runs on the target system. Which means every weird thing PASE does that's slightly different from AIX gets exercised, in production, all at once.

A small selection of what fell out:

uname -m returns the system serial number. Not ppc64. Not powerpc. The literal serial number of the machine — 007800001B91 on pub400. Go's build system uses uname -m to figure out what architecture it's running on. It does not know what to do with 007800001B91. Patch: hardcode gohostarch=ppc64 for ibmi, mirroring the existing AIX special case.

fstatat is a stub. Go's os package uses fstatat(dirfd, ...) to do stat-relative-to-a-directory operations. PASE's libc fstatat accepts exactly one value for dirfd: AT_FDCWD. Pass it any real directory file descriptor and it returns EINVAL. I found this out by writing — surprise — another C probe program, watching tests T1, T2, T8 pass and T3 through T7, T9, T10 fail with the same error code. The fix was to route ibmi onto Go's existing statat_other.go fallback path, the same code path used by plan9 and js/wasm — platforms that also don't have a real fstatat. The IBM i is in good company.

Errno isn't cleared on success. This one nearly cost me a full day of debugging the wrong thing. On every Unix you've ever used, a successful syscall doesn't touch errno. On PASE, a successful syscall also doesn't touch errno — but if a previous failed call left a stale value in there, that stale value is still there, and your test program will happily read it and conclude that the current call failed too. I had to add errno = 0 before every probe call. Lesson learned, applied to every probe written since.

The XCOFF loader races itself. This is the one I'm currently sitting on. PASE's dynamic loader is not thread-safe when multiple processes simultaneously resolve imports from the same shared library. Go's build system likes to do exactly that — it spawns parallel compile jobs, each of which forks and execs a fresh compile binary, all of which need libpthreads.a resolved at the same moment. About 50% of the time, two of them step on each other and the build dies with a loader error code. Workaround: pin cmd/go to single-threaded build (-p 1) on ibmi. It's slow but it's deterministic. A real fix is going to need either retry logic in the runtime's exec path or a pre-warm of the loader's cache.

There are more patches like this. I'll spare you them. The shape is always the same: write a probe, confirm what PASE actually does, write the smallest possible patch, document, move on.

Step 3: The Reference Port That Already Did Half The Work

About halfway through Phase 2 I went back to look at ibmigo's git history more carefully — Sumit’s port, the existing fork I'd decided not to use as my base — and found that he had independently discovered most of the same PASE quirks I was hitting. Same -p=1 workaround for the loader race. Same maxbg=1 for the GC. Even the same trick of retrying mkdir in a loop because PASE's syscalls are sometimes spuriously racy.

I'd been worried I was the only one seeing some of this. Turns out: no, this is just what PASE is. Sumit and I have shared confused experiences across years of disconnected effort, and we both ended up at almost identical workarounds. There is some comfort in that.

Common Europe Congress 2026 is there!

The agenda is published! Do you want to know where AIX is going to? It means you MUST visit the Common Europe Congress in Lyon, France. There will be sessions about new AIX features and open source community development. We will talk about AIX and IBM Power automation and Zero Downtime for AIX. Join me in Lyon!

Where We Are, And Where We're Going

The current state: cross-compile from my Mac to IBM i works cleanly. A native Go toolchain on IBM i builds, but only with GOMAXPROCS=1 and -p=1, and make.bash succeeds maybe half the time because of the loader race. I don't yet trust it enough to publish it. Alpha would be a generous label.

But it compiles things. It runs things. It is real Go, talking to real IBM i.

Once I'm confident the loader race is properly worked around — not just papered over — I'll publish the port. Anyone who wants to download the port and see what happens will be able to do so. After that comes node_exporter. And after that, the IBM i finally gets to join its AIX and Linux siblings in the same Prometheus dashboard, scraped on the same port 9100, looking like every other host on the network.

Wait a bit. You'll most probably get it.

Thanks

This work would have been considerably harder, and possibly impossible at this speed, without pub400.com — the free public IBM i system run by Holger Scherer. Every probe program in this article ran on pub400. Every "okay but does PASE actually do this?" question got answered there. Holger has been running this service for years, free of charge, for the IBM i community. If you're doing anything with IBM i and you don't have your own LPAR handy, pub400 is where you go. Holger, thank you.

Thanks also to Sumit Goyal, who built the ibmigo port and sent me the link when I started this. Several of the trickier PASE workarounds in my patch series — the loader-race mitigations especially — exist in something like their current form because Sumit got there first and his repo is public. Independent confirmation that you're not hallucinating a problem is worth a lot when you're three days into debugging a 50%-flaky build.

Share

Have fun porting software to your favorite platform!

Andrey

Hi, I am Andrey Klyachkin, IBM Champion and IBM AIX Community Advocate. This means I don’t work for IBM. Over the last twenty years, I have worked with many different IBM Power customers all over the world, both on-premise and in the cloud. I specialize in automating IBM Power infrastructures, making them even more robust and agile. I co-authored several IBM Redbooks and IBM Power certifications. I am an active Red Hat Certified Engineer and Instructor.

Follow me on LinkedIn, Twitter and YouTube.

You can meet me at events like IBM TechXchange, the Common Europe Congress, and GSE Germany’s IBM Power Working Group sessions.

Read more

Not on AIX, apparently. Not officially, anyway.

Read more

What could I answer? Yup, thanks for your birthday wishes! ;-)

Prometheus

No, it was not about the titan from Greek mythology, who helped people to steal fire from the gods.

It is about Prometheus monitoring software, which was started many years ago by SoundCloud, and is now a part of the Cloud Native Computing Foundation (CNCF). Prometheus is very popular in different clouds. It is easy to deploy, and paired with Grafana, it makes monitoring very visual and democratic.

No, Prometheus does not run on AIX.

If you wish to try an older version of Prometheus on AIX, you can do it. Several years ago, I ported some Prometheus versions to AIX, which you could find on my power-devops.com site. But the site doesn’t work right now.

The patches to the Prometheus code were not accepted because AIX is missing some functions available on Linux, and without these functions, Prometheus can’t guarantee that the data was written to the disk.

But Prometheus was available for Linux on IBM Power (ppc64le) for years.

Node Exporter

Prometheus works with so-called exporters that are installed on target servers.

The exporters are small HTTP (or HTTPS) servers that provide metrics - data about the target server.

Prometheus collects information from those exporters - or scrapes them.

There are exporters for specific services, and there are exporters for the operating system - node exporters.

Of course, it was not a big problem to compile the standard node exporter for AIX. I did it almost the same time, I ported Prometheus to AIX. But the node exporter didn’t collect any useful information about AIX.

This was the reason to start developing Google Go interface to AIX libperfstat. It is available freely on Github - https://github.com/power-devops/perfstat/.

I also tried to develop my own AIX node exporter, but never published it.

But now there is an official AIX node exporter from the Prometheus Team!

How to get the AIX node exporter

It is very easy!

• Open the node exporter Github repo

• On the right side, find “Releases” and click the latest release

• Now you should see the latest release and assets for it

• Click node_exporter-1.11.1.aix-ppc64.tar.gz (or what is the latest release when you read it) and download the file.

The archive contains only three files:

• LICENSE

• NOTICE

• node_exporter

What you need is the last file. You can copy it to the target system and run it there. There are no dependencies and no installation.

How to run the AIX node exporter

The easiest way to run it is simply to start it.

After you started it, it listens on port 9100, and you can check if it is working by using curl:

curl http://localhost:9100/metrics

You will get a long list with different metrics.

You don’t need any special knowledge to get through the list and check which metrics are delivered by the node exporter.

Of course, you don’t want to start the node exporter every time manually. You can simply add it to /etc/inittab like:

mkitab 'node_exporter:23456789:once:/usr/bin/node_exporter >/dev/console 2>&1

Now it starts every time with the system.

Common Europe Congress 2026 is there!

The agenda is published! Do you want to know where AIX is going to? It means you MUST visit the Common Europe Congress in Lyon, France. There will be sessions about new AIX features and open source community development. We will talk about AIX and IBM Power automation and Zero Downtime for AIX. Join me in Lyon!

Do you have any ideas how to make it better?

I didn’t do any thorough tests of the new node exporter. I only started it on AIX 7.3 TL3 and checked if it works.

It will definitely not work on POWER7. Because it is compiled with Google Go, and the Google Go compiler supports only POWER8 or newer CPUs.

It will probably not work on AIX 7.1, 6.1, or 5.3. I don’t have any AIX 5.3 to test, but if I have time, I will test it with AIX 6.1 or 7.1.

It probably doesn’t have all the metrics you need. It definitely has fewer metrics if you compare it to njmon.

If you have any ideas on what can be done better or any suggestions for metrics, let me know! Simply answer to the newsletter, if you’ve got it by email, or write a comment down. I will check and make a patch to enable the metric you need.

Share

Have fun with the AIX node exporter!

Andrey

Hi, I am Andrey Klyachkin, IBM Champion and IBM AIX Community Advocate. This means I don’t work for IBM. Over the last twenty years, I have worked with many different IBM Power customers all over the world, both on-premise and in the cloud. I specialize in automating IBM Power infrastructures, making them even more robust and agile. I co-authored several IBM Redbooks and IBM Power certifications. I am an active Red Hat Certified Engineer and Instructor.

Follow me on LinkedIn, Twitter and YouTube.

You can meet me at events like IBM TechXchange, the Common Europe Congress, and GSE Germany’s IBM Power Working Group sessions.

Read more

But there are some problems with Terraform and IBM Power.

First, it doesn’t run on IBM Power. You can download it for the x86 CPUs for almost every operating system. You can download it for ARM64 CPUs for Linux. You can even download it for Solaris. But you can’t find it for Linux on IBM Power (ppc64le) or for AIX.

OK, I ported it several times in the past, and you can find an older version of Terraform on my site.

But we have the second problem. Terraform works with plugins. Terraform itself is a configuration language plus a runner for the plugins. The plugins do the work.

No, there are no Terraform plugins to deploy an LPAR on IBM Power.

Or better to say, there was no Terraform plugin till today.

I think Friday the 13th is a good date to publish something new.

Let me introduce…

Terraform provider for IBM Power HMC

Here it is. You need Terraform for x86_64 (yes, no Power support right now), an IBM Power HMC, and the plugin. Then you write some code and deploy an LPAR.

What can it look like?

terraform {
  required_providers {
    ibmpower = {
      source  = "registry.terraform.io/power-devops/hmc"
      version = "0.0.1"
    }
    dns = {
      source = "hashicorp/dns"
    }
  }
}

variable "hmc_host" {
  description = "HMC hostname or IP address"
  type        = string
}

variable "hmc_username" {
  description = "HMC username"
  type        = string
}

variable "hmc_password" {
  description = "HMC password"
  type        = string
  sensitive   = true
}

variable "system_name" {
  description = "Managed system name where the LPAR will be created"
  type        = string
}

variable "lpar_name" {
  description = "Name of the LPAR to create"
  type        = string
  default     = "test-lpar"
}

provider "ibmpower" {
  host     = var.hmc_host
  username = var.hmc_username
  password = var.hmc_password
  insecure = true
}

data "ibmpower_vios_list" "all" {
  system_name = var.system_name
  sort = true
}

resource "ibmpower_lpar" "example" {
  system_name = var.system_name
  name        = var.lpar_name

  desired_mem = 8192
  desired_procs = 2
  sharing_mode    = "uncap"
  max_virtual_slots = 20
  boot_mode         = "norm"

  virtual_eth_adapters = [
    {
      is_required = true
      port_vlan_id = 1
    }
  ]

  virtual_fc_adapters = [
    {
       remote_lpar_id = data.ibmpower_vios_list.all.vios[0].lpar_id
       remote_slot_number = data.ibmpower_vios_list.all.vios[0].next_virtual_slot
       is_required = true
       physical_fcs = "fcs0"
    },
    {
       remote_lpar_id = data.ibmpower_vios_list.all.vios[1].lpar_id
       remote_slot_number = data.ibmpower_vios_list.all.vios[1].next_virtual_slot
       is_required = true
       physical_fcs = "fcs1"
    }
  ]
}

resource "ibmpower_lpar_action" "example_install" {
  system_name = ibmpower_lpar.example.system_name
  name = ibmpower_lpar.example.name
  action = "netboot"
  wait_for_state = "Running"
  timeout = 900
  netboot_server_ip   = "10.10.2.4"
  netboot_gateway_ip  = "10.10.2.1"
  netboot_client_ip   = "10.10.2.20"
  netboot_subnet_mask = "255.255.255.0"
}

resource "terraform_data" "wait_for_boot" {
  depends_on = [ibmpower_lpar_action.example_install]

  provisioner "local-exec" {
    command = <<-EOT
      for i in $(seq 1 120); do
        ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no ${var.lpar_name} "echo ready" 2>/dev/null && exit 0
        echo "Waiting for ${var.lpar_name}... ($i/120)"
        sleep 30
      done
      echo "Timeout waiting for ${var.lpar_name}"
      exit 1
    EOT
  }
}

What does it all mean?

If you have never worked with Terraform, you can check the Terraform tutorials to learn the basics of Terraform’s language.

If you worked with Terraform before and know how to use it, you will find the documentation for the plugin in the plugin’s package.

BUT. There is one gotcha with the plugin. It is not published on the official Terraform registry, and no, I don’t have plans to publish it there.

If you want to use it, use the link to download it directly.

You must create a file called terraformrc for your project with the following contents:

provider_installation {
  dev_overrides {
    "registry.terraform.io/power-devops/hmc" = "/home/myuser/dir-with-the-plugin"
  }
  direct {}
}

and export the environment variable:

export TF_CLI_CONFIG_FILE=/home/myuser/path/to/terraformrc

Then you can start terraform apply and the LPAR will be created.

But… I use NIM, and before I install an LPAR, I must create a NIM definition

So what? Let me introduce my second Terraform plugin.

Terraform provider for IBM AIX NIM

Yes, I use NIM too. I must also create a NIM definition for each AIX LPAR and assign resources to it.

Even more, I want to configure my NIM server using Terraform.

Now it is possible!

The NIM plugin is also available for download.

The same conditions apply. The documentation is in the archive.

You can add something like this to the previous code:

terraform {
  required_providers {
    ibmpower = {
      source  = "registry.terraform.io/power-devops/hmc"
      version = "0.0.1"
    }
    nim = {
      source  = "registry.terraform.io/power-devops/nim"
      version = "0.0.1"
    }
    dns = {
      source = "hashicorp/dns"
    }
  }
}

variable "nim_host" {
  description = "NIM hostname or IP address"
  type        = string
}

variable "nim_username" {
  description = "NIM username"
  type        = string
}

variable "nim_pkey_path" {
  description = "Path to the user's private key"
  type        = string
  sensitive   = true
}

provider "nim" {
  host             = var.nim_host
  username         = var.nim_username
  use_agent        = true
}

data "dns_a_record_set" "nim" {
  host = var.nim_host
}

resource "nim_standalone" "example" {
  name = ibmpower_lpar.example.name
  ssl = true
}

resource "nim_bos_inst" "example" {
  standalone = nim_standalone.example.name
  lpp_source = "7300-03-01-2520-lpp_source"
  spot = "7300-03-01-2520-spot"
  resolv_conf = "resolv_conf_default"
  bosinst_data = "bosinst_data_default"
  image_data = "image_data_default"
  fb_script = "fb_rootkeys"
}

resource "ibmpower_lpar_action" "example_install" {
  depends_on = [nim_bos_inst.example]
  system_name = ibmpower_lpar.example.system_name
  name = ibmpower_lpar.example.name
  action = "netboot"
  wait_for_state = "Running"
  timeout = 900
  netboot_server_ip   = "10.10.2.4"
  netboot_gateway_ip  = "10.10.2.1"
  netboot_client_ip   = "10.10.2.20"
  netboot_subnet_mask = "255.255.255.0"
}

Or if you wish to configure a new NIM server:

resource "nim_bosinst_data" "bos" {
  name = "bosinst_data_default"
  content = file("${path.module}/files/bosinst_data_default")
  destination = "/nim/rsrc/bosinst_data_default"
}

resource "nim_image_data" "img" {
  name = "image_data_default"
  content = file("${path.module}/files/image_data_default")
  destination = "/nim/rsrc/image_data_default"
}

resource "nim_resolv_conf" "resolv_conf_default" {
  name = "resolv_conf_default"
  content = file("${path.module}/files/resolv_conf_default")
  destination = "/nim/rsrc/resolv_conf_default"
}

resource "nim_script" "nim_pwreset" {
  name = "pwreset"
  content = file("${path.module}/files/pwreset.sh")
  destination = "/nim/scripts/pwreset"
}

resource "nim_script" "nim_authkeys" {
  name = "authkeys"
  content = file("${path.module}/files/authkeys.sh")
  destination = "/nim/scripts/authkeys"
}

resource "nim_fb_script" "nim_fbscript" {
  name = "fb_rootkeys"
  content = file("${path.module}/files/fb_rootkeys.sh")
  destination = "/nim/scripts/fb_rootkeys"
}

resource "nim_lpp_source" "_7300-03-01-2520-lpp_source" {
  name = "7300-03-01-2520-lpp_source"
  location = "/nim/lpp/7300-03-01-2520-lpp_source"
  source = "/nim/dist/AIX_v7.3_Install_7300-03-01-2520_flash_122025_LCD8265308.iso"
  iso_fs_type = "udfs"
}

resource "nim_spot" "_7300-03-01-2520-spot" {
  name = "7300-03-01-2520-spot"
  location = "/nim/spot"
  source = nim_lpp_source._7300-03-01-2520-lpp_source.name
}

Support the Power DevOps Newsletter!

If you like reading technical articles about IBM Power, AIX, and Linux on IBM Power, consider upgrading to the paid tier to show your support. As a paid subscriber, you not only get regular posts, but you will get additional posts with the full code and further explanations, access to the whole archive of the blog, and take part in our monthly calls where you can ask your questions and propose topics for future newsletters. Be an active member of our community!

Some final words

These are beta version software. You can use it for free, but there is no guarantee that it works in your environment. And of course, no liability. The providers provided “AS IS”.

The providers work with both Terraform and OpenTofu, the free implementation of Terraform.

If you have any problems using them, feel free to drop an email to me. Simply answer the newsletter, and the email should come to me. I will check if I can help you and answer as soon as I can do it.

If you need official support, do the same. Write an email to me, and we will discuss what I can do for you.

Because this is beta software and I am actively developing it, the archives can be updated without any notice. The version can be the same, but the providers will be different ;-)

No, I don’t plan to make the source code available.

No, I don’t know if the providers will always be freely available.

It always depends on how many customers I have and how many of them are interested in these plugins.

Anyway, you can download the plugins right now and use them! If you have downloaded them, nobody can take them away from your computer.

Share

Have fun deploying LPARs with Terraform!

Andrey

Hi, I am Andrey Klyachkin, IBM Champion and IBM AIX Community Advocate. This means I don’t work for IBM. Over the last twenty years, I have worked with many different IBM Power customers all over the world, both on-premise and in the cloud. I specialize in automating IBM Power infrastructures, making them even more robust and agile. I co-authored several IBM Redbooks and IBM Power certifications. I am an active Red Hat Certified Engineer and Instructor.

Follow me on LinkedIn, Twitter and YouTube.

You can meet me at events like IBM TechXchange, the Common Europe Congress, and GSE Germany’s IBM Power Working Group sessions.

Read more

Read more

Read more

Registration of NIM clients

Read more

Read more

Yes, nothing.

This is exactly my point. We are too few. I must know that you need what I write.

The playbook

Read more

Read more

Read more

But last week I wrote about image.data NIM resource, and I didn’t have enough space to post the whole file together with my explanations. I do it today.

image.data

Read more

Read more

Read more

Read more

As for me, 2025 was like a roller coaster. It had so many events, and they were so different, that I can’t even say which one was the best during the yea…

Read more

Read more