When troubleshooting OpenShift IPI (Installer-Provisioned Infrastructure) installations on IBM Cloud, you'll often need to check bootstrap logs. This guide walks you through SSH'ing into your VPC, using a jumphost, and accessing your bootstrap node to examine those critical logs.

note: Some commands may change over time, please refer to https://cloud.ibm.com/docs/vpc?topic=vpc-vsi_is_connecting_linux for the most up to date commands.

Preliminary Information:

What is a bootstrap?

• A bootstrap node is a temporary machine that OpenShift uses during the installation process.

   Think of it as the "construction supervisor" for your cluster. It:

• Coordinates the initial deployment of your cluster

• Hosts the control plane components temporarily

• Helps configure the master nodes

• Is destroyed automatically once the cluster is up and running

• When an IPI installation fails or hangs, the bootstrap node's logs contain vital clues about what went wrong.

What is a floating IP?

• A floating IP is a static, publicly accessible IP address that you can assign to resources in your VPC.

• Think of it as:

• A permanent address that doesn't change

• A bridge between the public internet and your private VPC

• Something you can move between different machines as needed

• For our purposes, you'll typically attach a floating IP to a jumphost or bastion server so you can access your VPC from the internet. 

What is an SSH key and how do I get my SSH key?

• • SSH (Secure Shell) keys are a pair of cryptographic keys used for secure authentication:

    • Private key: Stays on your local machine (never share this!)

    • Public key: Gets uploaded to IBM Cloud and placed on your VPC instances

  • For more information check out these articles:

    • https://cloud.ibm.com/docs/vpc?topic=vpc-ssh-keys&interface=ui

    • https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-ssh-keys

       

Checking if you already have an SSH key:

• On Linux/Mac:

ls -la ~/.ssh/

Look for files named id_rsa and id_rsa.pub (or id_ed25519 and id_ed25519.pub).

• • • On Windows (PowerShell):

ls ~\.ssh\

 

Creating a new SSH key:

• • •  On Linux/Mac:

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Ensure you are using your email that is associated with the IBM Cloud account you are installing the cluster.

• • •  On Windows (PowerShell or Git Bash):

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Press Enter to accept the default location, then optionally set a passphrase for added security.

Viewing your public key:

• • •  On Linux/Mac:

cat ~/.ssh/id_rsa.pub

• • •  On Windows (PowerShell):

type ~\.ssh\id_rsa.pub

You'll need to add this public key to IBM Cloud before you can SSH into your instances.

Note: SSH key should be added during the installation process when it prompts you to choose your ssh key.

Furthermore you can add your SSH to your IBMCloud account by going to infrastructure → Compute → sshKeys and click on “Create +” , here  you can add in the ssh key generated using the ssh-keygen -t rsa command above

How do I create my VPC jumphost?

• • Navigate to Infrastructure → Compute → Virtual Server Instances for VPC (https://cloud.ibm.com/infrastructure/compute/vs)

    Click on "Create +"

    Configuration details:

    • Location: Choose the same region/zone as your OpenShift cluster (the defaults should be correct)

    • Name: We recommend using a name that corresponds to your subnet for easier identification and teardown (e.g., "cluster-name-jumphost")

    • Resource Group: Select the same resource group used during your IPI installation

    • Image and profile is good as is

    • SSH Key: Choose the same SSH key you specified during the IPI installation process, this SSH key should be saved under SSH keys in the account check note above for adding SSH into your IBMCloud account

    • Storage (boot volume and data volumes) defaults are good.

    • Networking: Select the VPC associated with your cluster deployment (This will typically be named with your cluster name plus an installation-added prefix)

  • Finalize the creation by clicking on “Create Virtual Server” , in the Virtual private clouds page where all VPC are listed locate your recently created VPC click on it and under the instance overview click on the Networking tab. To add a floating IP to the network attachment click on the the VNIs ellipsis (the three dots … )  and click on edit floating IPs and attach a Floating IP, this is the IP address you will use later to ssh into the VPC.

What is a VPC?

• A VPC (Virtual Private Cloud) is your own isolated section of IBM Cloud's infrastructure.

• It's like having your own private network in the cloud where:

• You control the IP address ranges

• You define security rules (who can access what)

• Your resources are isolated from other customers

• You can create subnets, configure routing, and manage network access

• In the context of OpenShift installations, your cluster components (including the bootstrap node) live inside this VPC.

The Connection Journey: Understanding the Path

Before we dive into the commands, let's understand the journey your SSH connection takes:

Your Computer → Internet → Floating IP → Jumphost (in VPC) → Bootstrap Node

• • Your Computer: Where you start
  • Floating IP: The public gateway into your VPC
  • Jumphost: A bastion server inside your VPC with external access
  • Bootstrap Node: Your final destination (only accessible from within the VPC)

The bootstrap node typically doesn't have a public IP for security reasons, which is why we need the jumphost as an intermediary.

Prerequisites

Before you begin, ensure you have:

• IBM Cloud account with appropriate permissions
• IBM Cloud CLI installed (installation guide)
• Your SSH private key file (usually ~/.ssh/id_rsa)
• The OpenShift installation that created the bootstrap node
• Access to the VPC where your cluster is being deployed

Step 1: Gather Required Information

You'll need several pieces of information before connecting. Here's how to find them:

Option A: Using IBM Cloud Web Console

1. 1. 1. Log into IBM Cloud Console

      2. Navigate to Infrastructure → Network →  VPCs (the link is here: https://cloud.ibm.com/infrastructure/network/vpcs)

      3. Find your jumphost/bastion instance (note you might have to create the jump host, if so refer to the `How do I create my VPC Jump host section above`)

      4. Note the Floating IP address

      5. Find your bootstrap instance

      6. Note its GATEWAY IP address

Step 2: SSH into the Jumphost

Now that you have the floating IP of your jumphost, connect to it:

Basic SSH Command

ssh -i ~/.ssh/id_rsa vpcuser@<JUMPHOST_FLOATING_IP>

Explanation:

• • -i ~/.ssh/id_rsa: Specifies your private key file
  • root: The default username for RHCOS (Red Hat CoreOS) instances
  • <JUMPHOST_FLOATING_IP>: Replace with the actual floating IP created above when creating the VPC

 

Common Issues and Solutions

Issue: "Permission denied (publickey)"

• • Solution: Your public key may not be uploaded to IBM Cloud or the jumphost
    • Verify the key is correctly added to the instance

Issue: "Connection timed out"

• • Solution: Check that:
    • The floating IP is correctly assigned, you are ssh'ing using the correct floating IP (no typo)
    • Security group rules allow SSH (port 22) from your IP
    • Your network allows outbound SSH connections

 

Issue: "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED"

• • Solution: The host key has changed (common if you recreated the instance) bash
    • ssh-keygen -R <JUMPHOST_FLOATING_IP>
    • This removes the saved host identification to freshly retry the connecction

For Experienced Users: SSH Config File

To simplify connections, add this to ~/.ssh/config:

Host jumphost
    HostName <JUMPHOST_FLOATING_IP>
    User core
    IdentityFile ~/.ssh/id_rsa
    ServerAliveInterval 60

Then simply connect with:

ssh jumphost

Step 3: SSH from Jump Host to Bootstrap Node

Once connected to the jump host, you'll SSH again to reach the bootstrap node.

Add SSH key into the JumpHost:

Before you can jump to Bootstrap you need to add your SSH key into the ssh config file on the jumphost, this allows you to foward you ssh agent and use the jumphost as a proxy to reach the bootsrap

To SSH from the jumphost to the bootstrap, you have two options:

Option 1: SSH Agent Forwarding (Recommended - Easier & More Secure)

This method lets you use your local SSH key through the jumphost without copying it anywhere.

On your local machine (before connecting to jumphost):

# Start the SSH agent

eval "$(ssh-agent -s)"

# Add your SSH key to the agent

ssh-add ~/.ssh/<your id_rsa file name>

note: this does not end in .pub

# Verify your key was added (optional)

ssh-add -l

# Connect to jumphost with agent forwarding enabled (-A flag)

ssh -A -i ~/.ssh/id_rsa core@<JUMPHOST_FLOATING_IP>

Now from the jumphost, you can connect to bootstrap:

ssh core@<BOOTSTRAP_PRIVATE_IP>

The -A flag forwards your SSH credentials securely through the connection.

Option 2: Adding Your SSH Key to the Jumphost (Alternative Method)

If SSH agent forwarding doesn't work for your setup, you can add your public SSH key to the jumphost's authorized keys.

⚠️ Warning: This method is less secure and more complex. Option 1 is strongly recommended unless you have a specific reason to use this approach.

Connect to your jumphost:

ssh -i ~/.ssh/id_rsa core@<JUMPHOST_FLOATING_IP>

Check existing SSH keys on the jumphost:

cat ~/.ssh/authorized_keys

You should see at least one SSH key already there (added during VPC creation).

On your LOCAL machine (open a new terminal), copy your public key:

# View your public key

cat ~/.ssh/<id_rsa_file_name>.pub

note: If on mac try running ls -la ./.ssh this will display the saved ssh key names on your computer (note this includes private and public keys), identify which one is the one that you have associated with the IBM cloud and run the command above to copy the keys value. This file should be a .pub file since we want the public key to copy.

Copy the entire output (it should start with ssh-rsa or ssh-ed25519).

Back on the jumphost, add your key to authorized_keys ( there are two ways you can achieve this):

1. 1. 1. Method A - Using echo (easier for beginners):

echo "PASTE_YOUR_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys

1. 1. 1. Method B - Using vi editor:

vi ~/.ssh/authorized_keys 

• • • • Press Shift + G to go to the end of the file
      • Press o to create a new line and enter insert mode
      • Paste your public key
      • Press Esc to exit insert mode
      • Type :wq and press Enter to save and quit

Verify the key was added:

cat ~/.ssh/authorized_keys

You should now see your key listed.

Now you can SSH to the bootstrap

Connect to Bootstrap

ssh -AJ vpcuser@<JUMPHOST_FLOATING_IP> -i ~/.ssh/id_rsa-file-name core@<BOOTSTRAP_GATEWAY_IP>

Explanation:

• • • -A - Forwards your SSH agent so you can use your local SSH keys on remote machines
    • -J vpcuser@<JUMPHOST_FLOATING_IP> - Uses the jumphost as a proxy to reach the bootstrap node
    • The command connects through the jumphost to reach the bootstrap node (which isn't directly accessible)
    • Note: You may need to try different IP addresses to find the bootstrap node, as the exact IP can vary.
      • For example, if this doesn't work:

`ssh -AJ vpcuser@<jumpHost_IP> -i ~/.ssh/id_rsa_powervs core@192.168.0.11`

Try incrementing the last octect:

ssh -AJ vpcuser@<jumpHost_IP> -i ~/.ssh/id_rsa_powervs core@192.168.0.12 
ssh -AJ vpcuser@<jumpHost_IP> -i ~/.ssh/id_rsa_powervs core@192.168.0.13 

ssh -AJ vpcuser@<jumpHost_IP> -i ~/.ssh/id_rsa_powervs core@192.168.0.14

… and so on until you find the correct endpoint.

Once you hit the right endpoint you should see:

“key fingerprint is …. , The key is not known by other names, Are you sure you want to continue connecting (yes/no/[fingerprint])?”

Type ‘yes’ and you should be prompted with “This is the bootstrap node; it will be destroyed when the master is fully up”

This confirms you've successfully connected to the bootstrap node.

Important Notes:

• • • Use the Gateway IP of the bootstrap (not this is found in the Power Virtual Server → click on your workspace created → Virtual Sever Instances → look for VSI ending in ‘…. - bootstrap’
      • click on the bootstrap VSI to access its overview, click on the networking tab and note the Gateway IP
    • The bootstrap is only accessible from within the VPC
    • The jumphost should have access to your SSH key (either via ssh-agent forwarding or by having the key on the jumphost)
    • if you want more details when sshing using the command -vv this will give you ther verbose output of the ssh comand, i.g : ssh -vv  core@<boostrap_Gateway_IP>

Using SSH Agent Forwarding (Recommended) 

This allows you to use your local SSH key through the jumphost without copying it.

On your local machine (before connecting to jumphost):

# Start ssh-agent
eval "$(ssh-agent -s)"

# Add your key
ssh-add ~/.ssh/id_rsa

# Connect with agent forwarding enabled
ssh -A vpcuser@<JUMPHOST_FLOATING_IP>

Now from the jumphost, you can SSH to bootstrap without needing the key there:

ssh core@<BOOTSTRAP_GATEWAY_IP

Alternative: One-Line Command (ProxyJump)

You can jump directly to bootstrap in one command:

ssh -AJ vpcuser@<JUMPHOST_FLOATING_IP> core@<BOOTSTRAP_GATEWAY_IP>

Or add this to ~/.ssh/config:

Host bootstrap
    HostName <BOOTSTRAP_GATEWAY_IP>
    User core
    ProxyJump core@<JUMPHOST_FLOATING_IP>
    IdentityFile ~/.ssh/id_rsa

Then connect with:

ssh bootstrap

Step 4: Accessing Bootstrap Logs

Congratulations! You're now on the bootstrap node. Here's how to check the logs for your IPI build.

Key Locations and Commands

1. Check Bootstrap Service Status

sudo systemctl status bootkube

This shows the main bootstrap service that coordinates the installation.

Will also show you the activity of the bootstrap( i.e active or inactive (dead))

2. View Bootstrap Logs

journalctl -u bootkube -f

Explanation:

• • • journalctl: Command to view systemd logs
    • -u bootkube: Filter to the bootkube service
    • -f: Follow mode (like tail -f), shows new logs in real-time

Without follow mode (to see all logs):

journalctl -u bootkube --no-pager

3. Check Release Image Extraction

journalctl -u release-image -f

4. View All Bootstrap-Related Services

journalctl -b | grep bootstrap

5. Examine OpenShift Installation Logs

# Main installation directory
ls -la /opt/openshift/

# Manifest files
ls -la /opt/openshift/manifests/

# Check specific component logs
sudo journalctl -u kubelet -f

6. Check Container Logs

If you need to see logs from specific containers:

# List all containers
sudo podman ps -a

if podman doesn’t work try: sudo crictl ps -a

# View logs for a specific container
sudo podman logs <CONTAINER_ID>

if podman doesn’t work try: sudo crictl logs <CONTAINER_ID>

Common Log Patterns to Look For

Success indicators:

• • • "Bootstrap completed successfully"
    • "Waiting for control plane to come up"
    • "etcd cluster is healthy"

Warning signs:

• • • "Error:" or "Failed:" messages
    • Connection timeouts to API server
    • Certificate errors
    • Image pull failures

Saving Logs for Analysis

To save logs to a file for later analysis:

# Save bootstrap logs
journalctl -u bootkube --no-pager > /tmp/bootkube.log

# Copy to jumphost (from bootstrap)
scp /tmp/bootkube.log core@<JUMPHOST_FLOATING_IP>:~/

# Then from your local machine, copy from jumphost
scp core@<JUMPHOST_FLOATING_IP>:~/bootkube.log ./

Troubleshooting Common Issues

Can't SSH to Jumphost

1. 1. 1. Verify the floating IP is attached:

         • ibmcloud is instance <JUMPHOST_NAME>

      2. Check security group rules:

         •    ibmcloud is security-groups
   ibmcloud is security-group <SG_ID>

           Ensure there's a rule allowing inbound TCP on port 22 from your IP.

      3. Verify your public key is added:

         • Check in IBM Cloud Console under VPC Infrastructure → SSH keys

         • Ensure the key is associated with your instance

      4. If you are getting an Connection closed by UNKNOWN port 65535 error, ensure that:

         1. You followed the steps above to add your ssh key into the jump host known_keys file

         2. You added the key to SSH agent in your local computer, this is done by running the command: ssh-add ~/.ssh/<file_name> to which you should receive an expected output of Identity added: …..

         3. You may need to try different IP addresses to find the bootstrap node, as the exact IP can vary.: ssh -AJ vpcuser@<jumpHost_IP> -i ~/.ssh/id_rsa_powervs core@192.168.0.11 Try incrementing the last octet : ssh -AJ vpcuser@<jumpHost_IP> -i ~/.ssh/id_rsa_powervs core@192.168.0.12 … and so on keep trying 13,14,15.. ( see steps above on connecting to the bootstrap)

Can't SSH from Jumphost to Bootstrap

• • • Verify bootstrap is running:

# From jumphost
ping <BOOTSTRAP_GATEWAY_IP>

• • • Check SSH connectivity:

 nc -zv <BOOTSTRAP_GATEWAY_IP> 22

• • • Verify security group allows internal VPC traffic:

Security groups should allow traffic between instances in the VPC

Bootstrap Service Not Running

1. 1. 1. Check if the service exists:

sudo systemctl list-units | grep bootkube

1. 1. 1. Check for errors in service startup:

sudo systemctl status bootkube.service

1. 1. 1. Look at system logs: 

journalctl -xe

Best Practices

• Security

• Never share your private SSH key - Only your public key should be uploaded to IBM Cloud

• Use SSH agent forwarding instead of copying keys to the jumphost

• Restrict SSH access - Configure security groups to only allow SSH from known IP addresses

• Remove floating IPs when not needed to reduce attack surface

• Efficiency

• Set up SSH config file - Saves time and reduces errors

• Use screen or tmux on the jumphost - Keeps sessions alive if disconnected

• Create aliases for frequently used commands

• Log Analysis

• Save logs before bootstrap deletion - The bootstrap node is temporary

• Use grep effectively - Filter logs for specific errors or components

• Check timestamps - Correlate events across different log sources

Quick Reference: Command Cheat Sheet

# Connect to jumphost
ssh -i ~/.ssh/id_rsa vpcuser@<JUMPHOST_FLOATING_IP>

# Connect to jumphost with agent forwarding
ssh -A -i ~/.ssh/id_rsa vpcuser@<JUMPHOST_FLOATING_IP>

# From jumphost to bootstrap
ssh core@<BOOTSTRAP_GATEWAY_IP>

# Direct jump (one command)
ssh -J vpcuser@<JUMPHOST_FLOATING_IP> core@<BOOTSTRAP_GATEWAY_IP>

# View bootstrap logs (follow)
journalctl -u bootkube -f

# View bootstrap logs (all)
journalctl -u bootkube --no-pager

# Check bootstrap service status
sudo systemctl status bootkube

# View kubelet logs
sudo journalctl -u kubelet -f

# List containers
sudo pdoman ps -a

# View container logs
sudo podman logs <CONTAINER_ID>

# Save logs to file
journalctl -u bootkube --no-pager > /tmp/bootkube.log

Additional Resources

• IBM Cloud VPC Documentation

• OpenShift Installation Documentation

• IBM Cloud CLI Reference

• SSH Best Practices 

Conclusion

Accessing bootstrap logs through SSH might seem complex at first, but it follows a logical pattern: from your computer, through a jumphost, to the bootstrap node. Understanding this flow is crucial for troubleshooting OpenShift IPI installations on IBM Cloud.

Remember:

• • The bootstrap is temporary and will be deleted after successful installation
  • Security groups and SSH keys are critical for access
  • Logs contain valuable diagnostic information
  • Save important logs before the bootstrap node is destroyed

With this guide, you should be well quipped to debug, diagnose, and troubleshoot your OpenShift installations!