Underrated activity: microcode update
When, if not now?
Summer! Heat! Management wants me to drink more, and I do it. I bought a bottle of good Primitivo wine from Apulia (Italy), am listening to music, and fulfilling the wishes of my management. My children enjoy their summer holiday with their grandma in Sweden, much further north than this year’s Common Europe Congress, and I have time to reorganize my house.
I moved my working room to another floor and laid UTP cable to the DSL router. I also configured several FortiGates and Dell servers. I don’t know why some of you think that I only work with IBM Power, AIX, and Ansible. We all have to eat and earn some money for living. Last month, I automated Debian Linux installations on Microsoft Hyper-V, and yes, I used PowerShell. Am I proud of it? Not at all! But looking beyond my personal preferences helps me understand why I prefer to work with IBM Power and AIX.
This week I had a lot of fun with Dell servers. The Dell servers have iDRAC. Oh, gosh, it is so much better than IBM ASMI! The web interface is so lovely. You can configure so many options! I don’t think IBM will ever do ASMI so well as Dell iDRAC.
But. I set a new root password for iDRAC because the default one (calvin) is not secure at all. My password saving mechanism failed to save it. I don’t know why. It saved everything, but not the root’s password for iDRAC.
I had to reset it. I read many articles on the Internet, and the only way to reset it is to use a monitor and a local keyboard. Unfortunately, I don’t have a USB keyboard or a VGA monitor at home. Yes, you read it correctly. In 2025, I need a VGA monitor. Not HDMI, DVI, or USB. VGA!
I had to go to my old test lab and get my 15-year-old KVM switch to restore the iDRAC password on a modern Dell server.
My next failed step was to update the microcode on the adapters. Yes, I worked for many years with IBM Power, and I know that old microcode can have problems.
What would I do on IBM Power? Download new microcode.mic, run invscout, and get the list of new microcodes.
Dell? No way. Search for the adapters’ microcode manually, download it manually, and install it manually. Maybe there are other ways to do it, but I didn’t find them in iDRAC.
It brings me to the point that it is sometimes of advantage to see how other technologies work. Only to understand that IBM is not so backward. Or awkward. It is even very advanced.
But I still do the same on IBM Power manually!
Yes, it is still the manual task to download the microcode.mic and distribute it to the servers. Of course, it is easy to automate it using Ansible:
- name: Download new catalog
ansible.builtin.get_url:
url: https://public.dhe.ibm.com/software/server/firmware/catalog.mic
dest: /tmp/catalog.mic
delegate_to: localhost
run_once: true
become: false
- name: Copy catalog.mic to the target hosts
ansible.builtin.copy:
src: /tmp/catalog.mic
dest: /var/adm/invscout/microcode/catalog.mic
owner: root
group: system
mode: "0644"My only problem with the modern catalog.mic, is that it doesn’t contain systems’ firmware versions anymore. If you ignore the systems’ firmware output and can live with it, it is not the problem. Otherwise, search other ways, like using -t ibmwebsite in the HMC. But in this case, you must have an Internet connection.
Using HMC to update the microcode
What I love in the new HMC versions is that you don’t need any automation (almost) to update the adapters’ microcode. Select Update I/O firmware in the web interface, then select IBM website as the source, and that’s it!
Of course, it means you have Internet access from your HMC. The screen is, by the way, done from the HMC V11. Yes, I just clicked “Update” and got the new microcode.
After some time, everything is updated.
If you see the errors in the screenshot, they are only because we update the microcode on the adapter, not on each adapter’s port. All ports are already updated after we updated the first port on the adapter.
What to do if you don’t have a new HMC?
What’s the problem? Use Ansible!
Of course, it means you must manually download the adapters’ microcode and place it on an NFS share.
After you have downloaded the microcode, mount the NFS share with the code and install the RPM packages:
After the RPM packages with the microcode are installed, update it!
Are you in Switzerland, Austria, or Germany? Then this information is for you.
The following information is in German. If you are from the non-German-speaking part of Switzerland, please contact Hermann (see below, Anmeldung). If we have enough non-German-speaking participants, we will make the workshop in English. Or, we will think about making it one more time for English speakers.
Einladung zum ETS-Ansible Automation Workshop 2.0 für IBM Power Systems
Wann: 18.–19. September 2025
Wo: IBM Auditorium, IBM Schweiz AG, Vulkanstrasse 106, Postfach, 8010 Zürich
Beginn: 09:30 Uhr
Anmeldung: hermann.huber@ch.ibm.com
Kosten: Keine
🧑🏫 Trainer & Experten
Alexander Paul – IBM, Senior Technical Consultant
Andrey Klyachkin – IBM, Senior Technical Consultant
Michael Fiebig – IBM, Systems Engineer (Server Setup Lab Umgebung)
Stefan Gocke – IBM Business Partner
📋 Agenda
Tag 1 – Deployment & Automatisierung
Vormittag: Einführung und Grundlagen
Einführung in Ansible: Funktionsweise und Architektur
Installation von Ansible auf Linux & AIX
Vorbereitung der Ansible-Umgebung
Ansible Workspace einrichten
Notwendige Collections herunterladen
Zugriff auf Systeme prüfen (HMC, PowerVC, Storage, …)
Erstes Playbook: System-Ping
Unterschiede HMC vs. PowerVC für LPAR-Deployment
Nachmittag: LPAR Deployment & Automatisierung
LPAR Deployment: HMC oder PowerVC als Basis
OS-Installation (AIX, RHEL, SLES) und erste Konfiguration
LVM-Konfiguration: Volume-Gruppen, LVs, Filesysteme
Application Deployment: Oracle auf AIX oder Apache auf RHEL
OS-Tuning für spezifische Workloads
Ansible Automation Platform (AAP): Deployment über AAP
Herausforderungen und Unterschiede zur manuellen Ansible-Nutzung
Tag 2 – Betrieb & Performance-Tuning
Vormittag: Ansible im Betrieb
Warum Ansible für den Betrieb von Power-Systemen?
Netzwerk-Konfiguration & Performance-Tuning
Virtual Ethernet auf VIO/AIX/RHEL
Shared Ethernet Adapter (SEA)
SR-IOV für VIO/AIX/RHEL
Storage & LVM Performance
VSCSI-Setup und Optimierung
FC-HBA-Tuning für NPIV & Storage-Anbindung
Monitoring von pbuf, fsbuf & LVM-Konstrukten
Nachmittag: Updates & Security Management
Updates & Security Patching mit Ansible
VIO/AIX/RHEL Update-Prozesse
Security-Patching für VIO/AIX
Automatisierte Softwareverwaltung mit NIM
Best Practices & Troubleshooting
Typische Fehler und Lösungsansätze
PowerVC-Integration und mögliche Herausforderungen
Is it so easy to update the microcode as on x86?
As for me, it is even easier to update the adapters’ microcode on IBM Power than on x86. What I see almost daily is that x86 (Linux or Windows) administrators often overlook the adapters’ microcode. They ignore the updates and don’t pay much attention to them.
Is it good?
No, from my point of view, it is bad. I already solved some problems on the x86 servers just by updating the adapters’ microcode. It means the problems are there, they are simply ignored.
You may feel lucky that you manage IBM Power, where you have the HMC with the automatic microcode update, or can use Ansible to automate the updates.
Do it and never have an issue with the IBM support! ;-)
Have fun updating adapters’ microcode!
Andrey
Hi, I am Andrey Klyachkin, IBM Champion and IBM AIX Community Advocate. This means I don’t work for IBM. Over the last twenty years, I have worked with many different IBM Power customers all over the world, both on-premise and in the cloud. I specialize in automating IBM Power infrastructures, making them even more robust and agile. I co-authored several IBM Redbooks and IBM Power certifications. I am an active Red Hat Certified Engineer and Instructor.
Follow me on LinkedIn, Twitter and YouTube.
You can meet me at events like IBM TechXchange, the Common Europe Congress, and GSE Germany’s IBM Power Working Group sessions.