Help! All my disks are already in rootvg!

I don't have a spare disk for altinst_rootvg to upgrade my VIOS.

The problem is well known. To perform a VIOS upgrade, you need a spare disk. If you have a mirrored rootvg, you need two disks. The solution is well-known, too. Ask your storage administrator for additional disks! He or she will be very glad to hear it. Your manager? Too! What should we do? Many administrators, myself included, unmirror rootvg to get the second disk usable for the update.

Should we unmirror rootvg at all?

There is a risk. You (or your colleagues) mirrored the rootvg because they want to make the system reliable. If the SAS adapter or one of the two disks fails, nothing happens. VIOS works. Why should we reduce our reliability by unmirroring rootvg?

Because this is theory. In practice, you probably have two VIOSes. If the rootvg disk fails on one of the VIOSes, the second VIOS still works. But worse - if your rootvg disk on one of the VIOSes fails, probably all others will fail, too. Why? Because you bought all of them simultaneously, they are probably from the same order and have the same manufacturing date. They all have the same capabilities as MTBF (mean time between failures).

I worked once in a huge environment where we changed disks many times per day. We had many thousands of servers. They were bought in batches of 10s or 100s. They failed at the same time—the whole batch almost the same day.

On the other hand, think a little bit.

• How big is the risk that the disk will fail precisely when you do the upgrade?

• What does happen to the LPARs when one VIOS fails?

• Do you plan to reboot your VIOS during the upgrade?

Yes, the last question is mostly funny. You must reboot VIOS during the upgrade, and if the only rootvg disk fails during the procedure, nothing changes at all.

I don’t see any risk if you unmirror rootvg disks for the upgrade. After you finish the upgrade and see that everything works, you can mirror the disks again. I usually re-mirror them one week later.

Before we unmirror

A young AIX administrator's usual mistake is removing the boot disk from rootvg. It is possible, but then you must perform several more actions to restore the IPL device and the boot logical volume. Why should we do it? We can check the disk used to boot AIX or VIOS and remove another.

First, get the disk used to boot VIOS:

- name: Get boot volume
  ansible.builtin.command:
    cmd: bootinfo -b
  changed_when: false
  register: bootvol

Now let’s check which disks are in rootvg:

- name: Get physical volumes in rootvg
  ansible.builtin.command:
    cmd: lsvg -Lp rootvg
  changed_when: false
  register: vols

I use “-L” to avoid locking in case some LVM operation occurs while I query the volume group.

As the last step, we check all the volumes in rootvg and exclude our boot volume from the list:

- name: Set physical volumes list
  ansible.builtin.set_fact:
    volumes: "{{ volumes | default([]) + [item | split | first] }}"
  when: item | split | first != bootvol.stdout_lines.0
  loop: "{{ vols.stdout_lines[2:] }}"

If it looks cryptical for you, start from the last line. We loop through all volumes in the rootvg we’ve found. We saved the volumes in the variable vols, and the standard output of the lspv command is saved as vols.stdout_lines. The first line of the standard output is the header. We don’t need it, and start from the second line:

  loop: "{{ vols.stdout_lines[2:] }}"

Now, we go back to the beginning. We use the module ansible.builtin.set_fact to define a new variable called volumes. Because we didn’t define the variable before, we initialize it as an empty array:

volumes | default([])

Every time we iterate through the loop—you remember these lines of lspv output—we want to get the disk name and add it to the variable volumes. The disk name is always in the first column of the lspv output. To get it, we split the line and get the first word from it:

[item | split | first]

We use square brackets here because we add the value to the array. We don’t want to concatenate a string or make a mathematical number addition. We add the value to the end of the volumes array.

We repeat the action for all disks in the lspv output, but not for our boot volume, which is saved in the first line (number 0) of the bootinfo standard output:

when: item | split | first != bootvol.stdout_lines.0

Unmirroring rootvg

We are ready to unmirror our rootvg but only do it in one case. If we have something to unmirror:

- name: Perform unmirror
  when: volumes is defined and volumes | length > 0
  block:
    - name: Show volumes to remove from rootvg
      ansible.builtin.debug:
        msg: "The following volume will be removed from rootvg: {{ volumes.0 }}"

There are two ways to unmirror any volume group. You can use the command unmirrorvg or the command rmlvcopy.

The first is a shell script, which performs some checks and executes rmlvcopy.

The second is a command to remove a copy of a logical volume. I usually use this method.

To remove unneeded copies, we first must get a list of all logical volumes in rootvg:

    - name: Get logical volumes from rootvg
      ansible.builtin.command:
        cmd: lsvg -Ll rootvg
      changed_when: false
      register: lvols

    - name: Set logical volumes list
      ansible.builtin.set_fact:
        lvs: "{{ lvs | default([]) + [item | split | first] }}"
      loop: "{{ lvols.stdout_lines[2:] }}"

As you can see, I use the same trick for the volumes list. I issue lsvg -Ll rootvg to get the list of the logical volumes without obtaining LVM locks and then create the array lvs with the names of all logical volumes.

Now, we can remove all copies of these logical volumes:

    - name: Removing copies of logical volumes from rootvg
      ansible.builtin.command:
        cmd: "rmlvcopy {{ item }} 1 {{ volumes.0 }}"
      changed_when: true
      failed_when: false
      loop: "{{ lvs }}"

The command fails if a logical volume has only one or no copy on the target volume. I don’t want my playbook to fail and use failed_when to prevent failing.

We have removed all copies from the disk we want to use for our VIOS upgrade. In the perfect world, we can remove the disk from the rootvg. In my world, there can still be some logical volumes on the disk that were not mirrored. I use migratepv to move the rest to another disk.

    - name: Move the rest
      ansible.builtin.command:
        cmd: "migratepv {{ volumes.0 }} {{ bootvol.stdout_lines.0 }}"
      changed_when: true

The command doesn’t fail if there is nothing to move.

Now there should be nothing left on the disk, and we can remove it from rootvg:

    - name: Remove volume from rootvg
      ansible.builtin.command:
        cmd: "reducevg rootvg {{ volumes.0 }}"
      changed_when: true

Support the Power DevOps Newsletter!

Upgrade to our paid tier to unlock every article in the archive. Become a Founding Member for a little extra and book a 1-to-1 coffee chat with Andrey Klyachkin.

All done? Not really.

We must clean up LVM structures and re-create the bootloader.

Clean up the boot record from the removed volume first:

    - name: Clean boot
      ansible.builtin.command:
        cmd: "chpv -c {{ volumes.0 }}"

Then, clean up the LVM structures:

    - name: Clean LVM structure
      ansible.builtin.command:
        cmd: "chpv -C {{ volumes.0 }}"

One more action that can be done is to remove PVID from the volume:

    - name: Clean PVID
      ansible.builtin.command:
        cmd: "chdev -l {{ volumes.0 }} -a pv=clear"

If you don’t plan to upgrade the VIOS (or AIX) immediately after unmirror, ensure that the boot list contains the correct disk:

    - name: Set bootlist
      ansible.builtin.command:
        cmd: "bootlist -m normal {{ bootvol.stdout_lines.0 }}"

As the last action, recreate the BOS boot:

- name: Recreate BOS boot
  ansible.builtin.command:
    cmd: bosboot -a

If you removed the correct volume, the last command does not fail ;-)

Now, you have a spare volume for your future VIOS upgrade. Or an AIX upgrade. Because the procedure is the same.

Have fun unmirroring rootvg in an automated manner!

Andrey

P.S. We need you! Click the image and submit your presentation for the Common Europe Congress 2025 in Gothenburg, Sweden!

Hi, I am Andrey Klyachkin, IBM Champion and IBM AIX Community Advocate. This means I don’t work for IBM. Over the last twenty years, I have worked with many different IBM Power customers all over the world, both on-premise and in the cloud. I specialize in automating IBM Power infrastructures, making them even more robust and agile. I co-authored several IBM Redbooks and IBM Power certifications. I am an active Red Hat Certified Engineer and Instructor.

Follow me on LinkedIn, Twitter and YouTube.

You can meet me at events like IBM TechXchange, the Common Europe Congress, and GSE Germany’s IBM Power Working Group sessions.