Yes, I forgot to send it!

The last newsletter went to Nirvana.

Sometimes, you do everything you should do, everything is fine, and you relax into vacation or holidays. We had big holidays in Germany last weekend—from Friday to Monday. On Tuesday, I looked into my email inbox and found … nothing. Not completely nothing, but I didn’t see my last newsletter. What should I do? Send it one week later!

I hope you didn’t forget that we were in the middle of deploying a Kubernetes cluster on IBM Power. We started by downloading the software and creating our certification authority, then configured our future servers. The last newsletter was about configuring our control plane. What is missing? You are right! Worker nodes.

Still the same notes

The inventory is still the same:

---
all:
  children:
    jump:
      hosts:
        jumpbox:
          ansible_host: 10.11.0.1
    apiserver:
      hosts:
        server:
          ansible_host: 10.11.0.2
    workers:
      hosts:
        worker1:
          ansible_host: 10.11.0.11
        worker2:
          ansible_host: 10.11.0.12
  vars:
    ansible_user: root
    ansible_password: youKn0w1t!

It is still the same brain dump as the last three newsletters. Don’t expect too much. But it works in my environment ;-) I hope it will work for you, too. Let me know in the comments!

The last part is executed only on the nodes in the workers group.

Switch off swap

We start by switching off swap. We don’t need it on worker nodes, and our future kubelet service will not start if we don’t disable swap.

Swap can be enabled on worker nodes if you really need it. For many years, Kubernetes failed to start if swap was configured. But the latest versions (1.28, 1.32) introduced support for swap. You can reconfigure the kubelet service to work with swap.

First, we switch off our active swap areas if we have any. If we don’t have it, we ignore all errors:

    - name: Switch off swap
      ansible.builtin.command:
        cmd: swapoff -a
      failed_when: false

Next, we remove swap entries from /etc/fstab. I think you want to start Kubernetes even after restarting the Linux box. If you don’t remove the entries, the swap will automatically activate after the reboot, and Kubernetes will fail to start.

    - name: Remove swap from /etc/fstab
      ansible.builtin.mount:
        path: none
        fstype: swap
        state: absent_from_fstab

It is better to create special images for worker nodes without swap, but I use my standard images (you saw them already), and they have swap inside.

Copy and unpack the software

Before copying the software, we create the directories we need.

    - name: Create directories
      ansible.builtin.file:
        path: "{{ item }}"
        owner: root
        group: root
        mode: "0755"
        state: directory
      loop:
        - /etc/containerd
        - /etc/cni/net.d
        - /opt/cni/bin
        - /var/lib/kubelet
        - /var/lib/kube-proxy
        - /var/lib/kubernetes
        - /var/run/kubernetes

Now we copy the software we downloaded before on our jumpbox to the worker nodes:

    - name: Copy archives from jumpbox
      ansible.builtin.synchronize:
        src: "{{ item }}"
        dest: "{{ item }}"
      delegate_to: jumpbox
      loop:
        - /root/crictl-v1.31.1-linux-ppc64le.tar.gz
        - /root/containerd-2.0.0-linux-ppc64le.tar.gz
        - /root/cni-plugins-linux-ppc64le-v1.6.0.tgz
        - /root/runc.ppc64le
        - /root/kubectl
        - /root/kubelet
        - /root/kube-proxy

The last four files are simple binaries. We can copy them directly to the directories where they should be.

    - name: Copy binary files
      ansible.builtin.copy:
        src: "/root/{{ item }}"
        dest: "/usr/local/bin/{{ item }}"
        remote_src: true
      loop:
        - kubectl
        - kube-proxy
        - kubelet

    - name: Copy runc
      ansible.builtin.copy:
        src: /root/runc.ppc64le
        dest: /usr/local/bin/runc
        remote_src: true

The first three files, as the names imply, are archives. We must unpack them. All three archives are packed differently and must be unpacked to different places. That’s why we have three tasks—one task per archive.

    - name: Unpack crictl
      ansible.builtin.unarchive:
        dest: /usr/local/bin/
        src: /root/crictl-v1.31.1-linux-ppc64le.tar.gz
        remote_src: true

    - name: Unpack containerd
      ansible.builtin.unarchive:
        dest: /usr/local/
        src: /root/containerd-2.0.0-linux-ppc64le.tar.gz
        remote_src: true

    - name: Unpack cni-plugins
      ansible.builtin.unarchive:
        dest: /opt/cni/bin/
        src: /root/cni-plugins-linux-ppc64le-v1.6.0.tgz
        remote_src: true

The last task is to ensure that the unpacked files have the correct permissions:

    - name: Set permissions
      ansible.builtin.file:
        path: "{{ item }}"
        owner: root
        group: root
        mode: "0755"
      loop:
        - /usr/local/bin/crictl
        - /usr/local/bin/containerd
        - /usr/local/bin/ctr
        - /usr/local/bin/containerd-stress
        - /usr/local/bin/containerd-shim-runc-v2
        - /usr/local/bin/kubectl
        - /usr/local/bin/kube-proxy
        - /usr/local/bin/kubelet
        - /usr/local/bin/runc
        - /opt/cni/bin/vrf
        - /opt/cni/bin/firewall
        - /opt/cni/bin/macvlan
        - /opt/cni/bin/static
        - /opt/cni/bin/host-device
        - /opt/cni/bin/host-local
        - /opt/cni/bin/loopback
        - /opt/cni/bin/sbr
        - /opt/cni/bin/tuning
        - /opt/cni/bin/bridge
        - /opt/cni/bin/ptp
        - /opt/cni/bin/bandwidth
        - /opt/cni/bin/vlan
        - /opt/cni/bin/portmap
        - /opt/cni/bin/ipvlan
        - /opt/cni/bin/dummy
        - /opt/cni/bin/tap
        - /opt/cni/bin/dhcp

Container Network Interface

One of the archives we unpacked was CNI—container network interface. We should inform it of the networks we plan to use on our worker nodes. For me, everything is simple. I have worker1 and the network 10.200.1.0/24. On worker2, I have the network 10.200.2.0/24. As you see I can simply take the worker’s name, drop the word “worker” out of it, and get the network:

    - name: Set worker node subnet
      ansible.builtin.set_fact:
        worker_subnet: "10.200.{{ inventory_hostname | replace('worker', '') }}.0/24"

If you don’t have such a simple scheme, you can define host variables with the networks for each of your worker nodes.

The next step is to create configuration files for CNI:

    - name: Create 10-bridge.conf
      ansible.builtin.template:
        dest: /etc/cni/net.d/10-bridge.conf
        src: 10-bridge.conf.j2

    - name: Copy loopback.conf
      ansible.builtin.copy:
        dest: /etc/cni/net.d/99-loopback.conf
        src: 99-loopback.conf

This is the template 10-bridge.conf.j2:

{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "ranges": [
      [{"subnet": "{{ worker_subnet }}"}]
    ],
    "routes": [{"dst": "0.0.0.0/0"}]
  }
}

This is 99-loopback.conf:

{
  "cniVersion": "1.1.0",
  "name": "lo",
  "type": "loopback"
}

To enable our container network bridge, we must load the corresponding kernel module:

    - name: Add bridge netfilter module
      community.general.modprobe:
        name: br-netfilter
        state: present
        persistent: present

and configure two kernel parameters:

    - name: Set sysctls
      ansible.posix.sysctl:
        name: "{{ item }}"
        value: 1
        reload: true
        state: present
      loop:
        - net.bridge.bridge-nf-call-iptables
        - net.bridge.bridge-nf-call-ip6tables

Containerd configuration

This is the simplest task. We copy the configuration file for containerd, the container runtime we use.

    - name: Copy containerd-config.toml
      ansible.builtin.copy:
        dest: /etc/containerd/containerd-config.toml
        src: containerd-config.toml

Here is the file:

version = 2

[plugins."io.containerd.grpc.v1.cri"]
  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/opt/cni/bin"
  conf_dir = "/etc/cni/net.d"

Kubernetes configuration

Similar to containerd, we copy two more files. The first file is the configuration file for kubelet:

    - name: Copy kubelet-config.yaml
      ansible.builtin.template:
        dest: /var/lib/kubelet/kubelet-config.yaml
        src: kubelet-config.yaml.j2

Here is the template:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubelet/ca.crt"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.32.0.10"
cgroupDriver: systemd
containerRuntimeEndpoint: "unix:///var/run/containerd/containerd.sock"
podCIDR: "{{ worker_subnet }}"
resolvConf: "/etc/resolv.conf"
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/kubelet.crt"
tlsPrivateKeyFile: "/var/lib/kubelet/kubelet.key"

The second file is the configuration for kube-proxy:

    - name: Copy kube-proxy-config.yaml
      ansible.builtin.copy:
        dest: /var/lib/kube-proxy/kube-proxy-config.yaml
        src: kube-proxy-config.yaml

The configuration file:

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.200.0.0/16"

Automatic start of services on boot

We have finished configuring our Kubernetes cluster. Now, we must start it. We will do this by creating systemd unit files and starting them.

    - name: Copy systemd units
      ansible.builtin.copy:
        dest: "/etc/systemd/system/{{ item }}"
        src: "{{ item }}"
      loop:
        - containerd.service
        - kubelet.service
        - kube-proxy.service

I will not let you go without the files. If you need more information on how to write systemd unit files, let me know in the comments, and I will write about it.

This is containerd.service:

# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target dbus.service

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

This is kubelet.service:

[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

This is kube-proxy.service:

[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

After we created the unit files, we must let systemd know that they exist:

    - name: Reload systemd
      ansible.builtin.systemd_service:
        daemon_reload: true

The last step is to start the services and enable their autostart during the boot:

    - name: Start services
      ansible.builtin.systemd_service:
        name: "{{ item }}"
        enabled: true
        state: started
      loop:
        - containerd
        - kubelet
        - kube-proxy

Join me at Common Europe Congress 2025!

If you are in Europe, you are welcome to join me at the Common Europe Congress 2025. Even if you are not in Europe, you are still welcome!

The Common Europe Congress 2025 will be held from June 2nd to 5th in Gothenburg, Sweden, in a very beautiful place—Gothia Towers. Sweden is always beautiful, especially its nature. I like visiting Sweden, and I am there at least once every year.

The agenda for the Congress is already published. Last years it was the biggest IBM Power-related event in Europe. Just take a second and think about it. The biggest IBM Power event in Europe is organized by volunteers, not IBM. This is the reason to be there. This is not a marketing injection made by IBM, but real gurus and practitioners from all parts of Europe. You usually pay tons of money to get them to your site. The Congress costs just a fraction of their daily price, and you can ask your questions on all three days.

The early bird price is only 1020€ and valid till May 1st. If you are a member of a local Common country group, your price may even be lower (or zero). You can register here.

Remember to say hello to me in Sweden! By the way, “hello” in Swedish is “Hej.”

Your Kubernetes cluster is ready to use!

If you did everything correctly, your Kubernetes cluster is ready to use! You can deploy some applications to it. My usual test is to deploy nginx. Something like:

kubectl create deployment nginx --image=nginx:latest

should work without any problems. If kubectl can’t find the cluster, set the KUBECONFIG variable or specify the full path to your kubeconfig file:

KUBECONFIG=/etc/kubernetes/admin.conf kubectl create deployment nginx --image=nginx:latest

OK, using admin credentials is a bad idea. But we are testing only, aren’t we?

If you worked with OpenShift or Rancher, your new cluster doesn’t look like them. Many features are missing—no metrics, monitoring, or dashboard. This is a bare minimum Kubernetes cluster to learn how it works and gain the first experience.

Have fun with your new Kubernetes cluster!

Andrey

Hi, I am Andrey Klyachkin, IBM Champion and IBM AIX Community Advocate. This means I don’t work for IBM. Over the last twenty years, I have worked with many different IBM Power customers all over the world, both on-premise and in the cloud. I specialize in automating IBM Power infrastructures, making them even more robust and agile. I co-authored several IBM Redbooks and IBM Power certifications. I am an active Red Hat Certified Engineer and Instructor.

Follow me on LinkedIn, Twitter and YouTube.

You can meet me at events like IBM TechXchange, the Common Europe Congress, and GSE Germany’s IBM Power Working Group sessions.