Automating an OpenShift 4.7 install on bare metal with static IPs (part 1)

As an OpenShift consultant, I have often come across the requirement to install OCP on bare metal servers. And also quite often, this is combined with the absence of a DHCP server in the environment.

Up until OCP 4.5, one way to define a static network configuration was to catch the GRUB prompt from the console during booting of the RHCOS ISO and append the necessary kernel arguments, which is a challenging and error-prone procedure. Another, more complicated way, was to create a custom ISO, typically with multiple grub menu entries, one for each node of the cluster.

Starting from version 4.6, the RHCOS ISO switched to a live environment which boots to a shell prompt of a full RHCOS system running from memory.

RHCOS Live environment

Using nmcli or nmtui, we can configure networking for the live environment. Then we can use the new coreos-installer command with the--copy-network option which will install the system copying the network configuration from the live environment to the final RHCOS installation on disk.
This is a huge improvement over the previous installer but I thought we can further customize and automate the procedure. The idea (inspired by this blog post) is to prepare a custom RHCOS ISO image for each of the machines that will form our cluster, including the temporary bootstrap node. This ISO image will take care of automating the provisioning of the RHCOS machines, reducing the possibility for errors and also the time needed.

So let’s get into the details:

Bootstrap node

We are going to create two different Ignition configs:

• One to configure the live system and automate the installation (auto-bootstrap.ign)
• A second, intermediate one to configure the installed system on first boot from hard disk (bootstrap-int.ign)

Creating an Ignition file is a two-step process (as explained here). First we create a YAML-formatted Butane file. Then we run the Butane transpiler to convert the YAML file into a JSON Ignition config.

Let’s first take a look at the second file (bootstrap-int.yaml):

bootstrap-int.yaml

This file will be used to configure the installed system on first boot from hard disk. It points to the main Ignition config (created with the openshift-install create ignition-configs command) which is typically hosted on a web server (line 6). In this file we can also configure the hostname (lines 9–13) and NTP using chronyd (lines 14–23).

We will convert this file to an Ignition config by running the Butane utility as a container using podman on a RHEL 8 host:

podman run -i --rm quay.io/coreos/butane:release --pretty --strict < bootstrap-int.yaml > bootstrap-int.ign

We can now include the contents of bootstrap-int.ign in the first Ignition file which we’ll use to configure the live system and automate the installation:

auto-bootstrap.yaml

As we can see in lines 40–76, the bootstrap-int.ign will be created in the live environment’s filesystem at /home/core/bootstrap-int.ign. Static network configuration is included in lines 5–20. Automation is accomplished using a systemd service (lines 79–96) which calls a Bash script (lines 21–39). The script runs the actual installation (line 29) using bootstrap-int.ign as Ignition config and copying the static network configuration. If the installation is successful, the script will wait for 15 seconds and then exit with a zero (0) status which will cause the systemd service to poweroff the machine.

At this point we can remove the RHCOS ISO and power-on the machine again. This time the on-disk RHCOS installation will take over (with our custom network, hostname and chronyd configuration), pulling the final bootstrap.ign Ignition from the web server which will build the bootstrap node for our cluster.

But how do we start the live environment with our custom Ignition config in the first place?

First we need to convert auto-bootstrap.yaml to an Ignition config:

podman run -i --rm quay.io/coreos/butane:release --pretty --strict < auto-bootstrap.yaml > auto-bootstrap.ign

Then we use the coreos-installer binary (downloaded from here) to embed auto-bootstrap.ign into the original RHCOS ISO image (downloaded from here):

coreos-installer iso ignition embed -i auto-bootstrap.ign -o rhcos-4.7.0-auto.iso rhcos-4.7.0-x86_64-live.x86_64.iso

We finally use the rhcos-4.7.0-auto.iso created to boot our machine which will trigger the whole automated process.

Master and Worker nodes

For the master and worker nodes the process is similar. The difference is that we don’t need to create an intermediate Ignition config. We can use directly the master.ign and worker.ign files created with the openshift-install create ignition-configs command since they are very small and simple. This also means that we don’t need to host the master.ign and worker.ign files to a web server.

So here is a base Ignition config for automating a master installation:

auto-master.yaml

This example uses the original master.ign file with the addition of hostname and chronyd configuration (lines 65–84).

We can create the individual config files for each master/worker by replicating the file above and changing IPs and hostnames accordingly.
Finally, we can convert the created YAML files to Ignition configs and embed those to individual ISO images for each master/worker machine.

Wrap up

With the above process we can prepare custom RHCOS ISO images for each of our cluster’s nodes. These images can be used to automate the installation of OpenShift 4.7 on bare metal using static network configuration.

Now head over to part 2 to find out how we can further improve this!