Overcloud Administration¶
Updating Packages¶
It is possible to update packages on the overcloud hosts.
Package Repositories¶
If using custom DNF package repositories on CentOS or Rocky, it may be
necessary to update these prior to running a package update. To do this, update
the configuration in ${KAYOBE_CONFIG_PATH}/dnf.yml and run the following
command:
(kayobe) $ kayobe overcloud host configure --tags dnf --kolla-tags none
Package Update¶
To update one or more packages:
(kayobe) $ kayobe overcloud host package update --packages <package1>,<package2>
To update all eligible packages, use *, escaping if necessary:
(kayobe) $ kayobe overcloud host package update --packages "*"
To only install updates that have been marked security related:
(kayobe) $ kayobe overcloud host package update --packages "*" --security
Note that these commands do not affect packages installed in containers, only those installed on the host.
Kernel Updates¶
If the kernel has been updated, you will probably want to reboot the hosts to boot into the new kernel. This can be done using a command such as the following:
(kayobe) $ kayobe overcloud host command run --command "shutdown -r" --become
It is normally best to apply this to control plane hosts in batches to avoid
clustered services from losing quorum. This can be achieved using the
--limit argument, and ensuring services are fully up after rebooting before
proceeding with the next batch.
Running Commands¶
It is possible to run a command on the overcloud hosts:
(kayobe) $ kayobe overcloud host command run --command "<command>"
For example:
(kayobe) $ kayobe overcloud host command run --command "service docker restart"
To execute the command with root privileges, add the --become argument.
Adding the --verbose argument allows the output of the command to be seen.
Reconfiguring Containerised Services¶
When configuration is changed, it is necessary to apply these changes across the system in an automated manner. To reconfigure the overcloud, first make any changes required to the configuration on the Ansible control host. Next, run the following command:
(kayobe) $ kayobe overcloud service reconfigure
In case not all services’ configuration have been modified, performance can be
improved by specifying Ansible tags to limit the tasks run in kayobe and/or
kolla-ansible’s playbooks. This may require knowledge of the inner workings of
these tools but in general, kolla-ansible tags the play used to configure each
service by the name of that service. For example: nova, neutron or
ironic. Use -t or --tags to specify kayobe tags and -kt or
--kolla-tags to specify kolla-ansible tags. For example:
(kayobe) $ kayobe overcloud service reconfigure --tags config --kolla-tags nova,ironic
Deploying Updated Container Images¶
A common task is to deploy updated container images, without configuration changes. This might be to roll out an updated container OS or to pick up some package updates. This should be faster than a full deployment or reconfiguration.
To deploy updated container images:
(kayobe) $ kayobe overcloud service deploy containers
Note that if there are configuration changes, these will not be applied using
this command so if in doubt, use a normal kayobe overcloud service deploy.
In case not all services’ containers have been modified, performance can be
improved by specifying Ansible tags to limit the tasks run in kayobe and/or
kolla-ansible’s playbooks. This may require knowledge of the inner workings of
these tools but in general, kolla-ansible tags the play used to configure each
service by the name of that service. For example: nova, neutron or
ironic. Use -t or --tags to specify kayobe tags and -kt or
--kolla-tags to specify kolla-ansible tags. For example:
(kayobe) $ kayobe overcloud service deploy containers --kolla-tags nova,ironic
Upgrading Containerised Services¶
Containerised control plane services may be upgraded by replacing existing containers with new containers using updated images which have been pulled from a registry or built locally. If using an updated version of Kayobe or upgrading from one release of OpenStack to another, be sure to follow the kayobe upgrade guide. It may be necessary to upgrade one or more services within a release, for example to apply a patch or minor release.
To upgrade the containerised control plane services:
(kayobe) $ kayobe overcloud service upgrade
As for the reconfiguration command, it is possible to specify tags for Kayobe and/or kolla-ansible:
(kayobe) $ kayobe overcloud service upgrade --tags config --kolla-tags keystone
Running Prechecks¶
Sometimes it may be useful to run prechecks without deploying services:
(kayobe) $ kayobe overcloud service prechecks
As for other similar commands, it is possible to specify tags for Kayobe and/or kolla-ansible:
(kayobe) $ kayobe overcloud service upgrade --tags config --kolla-tags keystone
Stopping the Overcloud Services¶
Note
This step will stop all containers on the overcloud hosts.
To stop the overcloud services:
(kayobe) $ kayobe overcloud service stop --yes-i-really-really-mean-it
It should be noted that this state is persistent - containers will remain stopped after a reboot of the host on which they are running.
It is possible to limit the operation to particular hosts via
--kolla-limit, or to particular services via --kolla-tags. It is also
possible to avoid stopping the common containers via --kolla-skip-tags
common. For example:
(kayobe) $ kayobe overcloud service stop –kolla-tags glance,nova –kolla-skip-tags common
Destroying the Overcloud Services¶
Note
This step will destroy all containers, container images, volumes and data on the overcloud hosts.
To destroy the overcloud services:
(kayobe) $ kayobe overcloud service destroy --yes-i-really-really-mean-it
Deprovisioning The Cloud¶
Note
This step will power down the overcloud hosts and delete their nodes’ instance state from the seed’s ironic service.
To deprovision the overcloud:
(kayobe) $ kayobe overcloud deprovision
Saving Overcloud Service Configuration¶
It is often useful to be able to save the configuration of the control plane services for inspection or comparison with another configuration set prior to a reconfiguration or upgrade. This command will gather and save the control plane configuration for all hosts to the Ansible control host:
(kayobe) $ kayobe overcloud service configuration save
The default location for the saved configuration is $PWD/overcloud-config,
but this can be changed via the output-dir argument. To gather
configuration from a directory other than the default /etc/kolla, use the
node-config-dir argument.
Generating Overcloud Service Configuration¶
Prior to deploying, reconfiguring, or upgrading a control plane, it may be
useful to generate the configuration that will be applied, without actually
applying it to the running containers. The configuration should typically be
generated in a directory other than the default configuration directory of
/etc/kolla, to avoid overwriting the active configuration:
(kayobe) $ kayobe overcloud service configuration generate --node-config-dir /path/to/generated/config
The configuration will be generated remotely on the overcloud hosts in the
specified directory, with one subdirectory per container. This command may be
followed by kayobe overcloud service configuration save to gather the
generated configuration to the Ansible control host.
Validating Overcloud Service Configuration¶
Issues can arise in Kolla Ansible deployments when incorrect options are used
in configuration files. This is because OpenStack services will ignore unknown
options. It is also important to keep on top of deprecated options that may be
removed in the future. The oslo-config-validator can be used to check both
of these. This command will run it on the OpenStack control plane services:
(kayobe) $ kayobe overcloud service configuration validate --output-dir /path/to/save/results
Performing Database Backups¶
Database backups can be performed using the underlying support in Kolla Ansible.
In order to enable backups, enable Mariabackup in
${KAYOBE_CONFIG_PATH}/kolla.yml:
kolla_enable_mariabackup: true
To apply this change, use the kayobe overcloud service reconfigure command.
To perform a full backup, run the following command:
kayobe overcloud database backup
Or to perform an incremental backup, run the following command:
kayobe overcloud database backup --incremental
Further information on backing up and restoring the database is available in the Kolla Ansible documentation.
Performing Database Recovery¶
Recover a completely stopped MariaDB cluster using the underlying support in Kolla Ansible.
To perform recovery run the following command:
kayobe overcloud database recover
Or to perform recovery on specified host, run the following command:
kayobe overcloud database recover --force-recovery-host <host>
By default the underlying kolla-ansible will automatically determine which host to use, and this option should not be used.
Gathering Facts¶
The following command may be used to gather facts for all overcloud hosts, for both Kayobe and Kolla Ansible:
kayobe overcloud facts gather
This may be useful to populate a fact cache in advance of other operations.
