Troubleshooting¶

Firewalling¶

Due to the nature of firewall settings and customizations, bifrost does not change any local firewalling on the node. Users must ensure that their firewalling for the node running bifrost is such that the nodes that are being booted can connect to the following ports:

67/UDP for DHCP requests to be serviced
69/UDP for TFTP file transfers (Initial iPXE binary)
6301/TCP for the ironic API
8080/TCP for HTTP File Downloads (iPXE, Ironic-Python-Agent)

If you encounter any additional issues, use of tcpdump is highly recommended while attempting to deploy a single node in order to capture and review the traffic exchange between the two nodes.

NodeLocked Errors¶

This is due to node status checking thread in ironic, which is a locking action as it utilizes IPMI. The best course of action is to retry the operation. If this is occurring with a high frequency, tuning might be required.

Example error:

NodeLocked: Node 00000000-0000-0000-0000-046ebb96ec21 is locked by
host $HOSTNAME, please retry after the current operation is completed.

New image appears not to be deploying upon deployment¶

When deploying a new image with the same previous name, it is necessary to purge the contents of the TFTP master_images folder which caches the image file for deployments. The default location for this folder is /tftpboot/master_images.

Additionally, a playbook has been included that can be used prior to a re-installation to ensure fresh images are deployed. This playbook can be found at playbooks/cleanup-deployment-images.yaml.

Building an IPA image¶

Troubleshooting issues involving IPA can be time consuming. The IPA developers HIGHLY recommend that users build their own custom IPA images in order to inject things such as SSH keys, and turn on agent debugging which must be done in a custom image as there is no mechanism to enable debugging via the kernel command line at present.

Custom IPA images can be built a number of ways, the most generally useful mechanism is with diskimage-builder as the distributions typically have better hardware support than CoreOS and Tiny Core Linux. However, CoreOS and Tiny Core based images are what are used by the OpenStack CI for ironic tests.

CoreOS::

https://opendev.org/openstack/ironic-python-agent/src/branch/master/imagebuild/coreos/README.rst

TinyIPA::

https://opendev.org/openstack/ironic-python-agent/src/branch/master/imagebuild/tinyipa/README.rst

For documentation on diskimage-builder, See::

https://docs.openstack.org/diskimage-builder/latest/.

It should be noted that the steps for diskimage-builder installation and use to create an IPA image for Bifrost are the same as for ironic. See: https://docs.openstack.org/ironic/latest/install/deploy-ramdisk.html

This essentially boils down to the following steps:

1. git clone https://opendev.org/openstack/ironic-python-agent
2. cd ironic-python-agent
3. pip install -r ./requirements.txt #. If you don’t already have docker installed, execute: sudo apt-get install docker docker.io
4. cd imagebuild/coreos
5. Edit oem/cloudconfig.yml and add --debug to the end of the ExecStart setting for the ironic-python-agent.service unit.
6. Execute make to complete the build process.

Once your build is completed, you will need to copy the images files written to the UPLOAD folder, into the /httpboot folder. If your utilizing the default file names, executing cp UPLOAD/* /httpboot/ should achieve this.

Since you have updated the image to be deployed, you will need to purge the contents of /tftpboot/master_images for the new image to be utilized for the deployment process.

Unexpected/Unknown failure with the IPA Agent¶

Many failures due to the IPA agent can be addressed by building a custom IPA Image. See Building an IPA image for information on building your own IPA image.

https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/devuser

export DIB_DEV_USER_USERNAME=customuser
export DIB_DEV_USER_PWDLESS_SUDO=yes
export DIB_DEV_USER_AUTHORIZED_KEYS=$HOME/.ssh/id_rsa.pub
disk-image-create -o /path/to/custom-ipa debian ironic-agent devuser

ssh_public_key_path is not valid¶

Bifrost requires that the user who executes bifrost have an SSH key in their user home, or that the user defines a variable to tell bifrost where to identify this file. Once this variable is defined to a valid file, the deployment playbook can be re-run.

ansible-playbook -i inventory/bifrost_inventory.py deploy-dynamic.yaml -e ssh_public_key_path=~/path/to/public/key/id_rsa.pub

Changing from TinyIPA to CoreOS IPA, or any other IPA Image¶

With-in the Newton cycle, the default IPA image for Bifrost was changed to TinyIPA, which is based on Tiny Core Linux. This has a greatly reduced boot time for testing, however should be expected to have less hardware support. If on a fresh install, or a re-install, you wish to change to CoreOS or any other IPA image, you will need to take the following steps:

1. Remove the existing IPA image ipa.vmlinuz and ipa.initramfs.
2. Edit the playbooks/roles/bifrost-ironic-install/defaults/main.yml file and update the ipa_kernel_upstream_url and ipa_kernel_upstream_url settings to a new URL. For CoreOS, these urls would be, https://tarballs.openstack.org/ironic-python-agent/coreos/files/coreos_production_pxe.vmlinuz and https://tarballs.openstack.org/ironic-python-agent/coreos/files/coreos_production_pxe_image-oem.cpio.gz respectively.
3. Execute the installation playbook, and the set files will be automatically downloaded again. If the files are not removed prior to (re)installation, then they will not be replaced. Alternatively, the files can just be directly replaced on disk. The default where the kernel and ramdisk are located is in /httboot/.