Team and repository tags¶

Prep¶

Validations that are run on a fresh machine before the undercloud is installed.

• 512e: Advanced Format 512e Support
• undercloud-cpu: Verify undercloud fits the CPU core requirements
• undercloud-disk-space: Verify undercloud fits the disk space requirements
• undercloud-ram: Verify the undercloud fits the RAM requirements

Pre Introspection¶

Validations that are run when the undercloud is ready to perform hardware introspection.

• check-network-gateway: Check network_gateway on the provisioning network
• ctlplane-ip-range: Check the number of IP addresses available to overcloud nodes
• dhcp-introspection: DHCP on the Introspection Network
• undercloud-cpu: Verify undercloud fits the CPU core requirements
• undercloud-disk-space: Verify undercloud fits the disk space requirements
• undercloud-neutron-sanity-check: Undercloud Neutron Sanity Check
• undercloud-ram: Verify the undercloud fits the RAM requirements
• undercloud-tokenflush: Verify token_flush is enabled in keystone users crontab.

Pre Deployment¶

Validation that are run right before deploying the overcloud.

• 512e: Advanced Format 512e Support
• ceilometerdb-size: Ceilometer Database Size Check
• deployment-images: Verify existence of deployment images.
• dhcp-provisioning: DHCP on the Provisioning Network
• network-environment: Validate the Heat environment file for network configuration
• undercloud-debug: Undercloud Services Debug Check
• undercloud-process-count: Check the number of OpenStack processes on undercloud

Post Deployment¶

Validations that are run after the overcloud deployment finished.

• ceph-health: Check the status of the ceph cluster
• controller-token: Unnamed
• controller-ulimits: Check controller ulimits
• haproxy: HAProxy configuration
• mysql-open-files-limit: MySQL Open Files Limit
• neutron-sanity-check: Neutron Sanity Check
• no-op-firewall-nova-driver: Verify NoOpFirewallDriver is set in Nova
• ntpstat: Verify all deployed nodes have their clock synchronised.
• openstack-endpoints: Check connectivity to various OpenStack services
• pacemaker-status: Check the status of the pacemaker cluster
• rabbitmq-limits: Rabbitmq limits
• stonith-exists: stonith-exists

Pre Update¶

Validations that are run right before a minor update of the undercloud or overcloud.

Pre Upgrade¶

Validations that are run right before a major upgrade of the undercloud or overcloud.

• check-repo-availability: Check availability of current repositories
• openstack-endpoints: Check connectivity to various OpenStack services
• stack-health: Stack Health Check
• undercloud-disk-space-pre-upgrade: Verify undercloud fits the disk space requirements to perform an upgrade
• undercloud-ram: Verify the undercloud fits the RAM requirements
• undercloud-service-status: Verify undercloud services state before running update or upgrade

Directory Structure¶

All validations are located in the validations directory. It contains a couple of subdirectories:

• the files directory contains scripts that are directly executable;
• the library one is for custom Ansible modules available to the validations;
• tasks is for common steps that can be shared between the validations.

Here is what the tree looks like:

validations
├── first_validation.yaml
├── second_validation.yaml
├── files
│   └── some_script.sh
├── library
│   ├── another_module.py
│   └── some_module.py
└── tasks
    └── some_task.yaml

Sample Validation¶

Each validation is an Ansible playbook with a known location and some meta-data. Here is what a minimal validation would look like:

---
- hosts: overcloud
  vars:
    metadata:
      name: Hello World
      description: This validation prints Hello World!
  tasks:
  - name: Run an echo command
    command: echo Hello World!

It should be saved as validations/hello_world.yaml.

As shown here, the validation playbook requires three top-level directives: hosts, vars -> metadata and tasks.

hosts specify which nodes to run the validation on. Based on the hosts.sample structure, the options can be all (run on all nodes), undercloud, overcloud (all overcloud nodes), controller and compute.

The vars section serves for storing variables that are going to be available to the Ansible playbook. The validations API uses the metadata section to read each validation’s name and description. These values are then reported by the API and shown in the UI.

The validations can be grouped together by specifying a groups metadata. Groups function similar to tags and a validation can thus be part of many groups. Here is, for example, how to have a validation be part of the pre-deployment and hardware groups:

metadata:
  groups:
    - pre-deployment
    - hardware

tasks contain a list of Ansible tasks to run. Each task is a YAML dictionary that must at minimum contain a name and a module to use. Module can be any module that ships with Ansible or any of the custom ones in the library subdirectory.

The Ansible documentation on playbooks provides more detailed information.

Ansible Inventory¶

ansible-playbook -i /usr/bin/tripleo-ansible-inventory validations/hello_world.yaml

[undercloud]
undercloud.example.com

[overcloud:children]
controller
compute

[controller]
controller.example.com

[compute]
compute-1.example.com
compute-2.example.com

[all:vars]
ansible_ssh_user=stack
ansible_sudo=true

Custom Modules¶

In case the available Ansible modules don’t cover your needs, it is possible to write your own. Modules belong to the validations/library directory.

Here is a sample module that will always fail:

#!/usr/bin/env python

from ansible.module_utils.basic import *

if __name__ == '__main__':
    module = AnsibleModule(argument_spec={})
    module.fail_json(msg="This module always fails.")

Save it as validations/library/my_module.py and use it in a validation like so:

tasks:
...  # some tasks
- name: Running my custom module
  my_module:
...  # some other tasks

The name of the module in the validation my_module must match the file name (without extension): my_module.py.

The custom modules can accept parameters and do more complex reporting. Please refer to the guide on writing modules in the Ansible documentation.

Learn more at the Ansible documentation page about writing custom modules.

Running a validation¶

Running the validations require ansible and a set of nodes to run them against. These nodes need to be reachable from the operator’s machine and need to have an account it can ssh to and perform passwordless sudo.

The nodes need to be present in the static inventory file or available from the dynamic inventory script depending on which one the operator chooses to use. Check which nodes are available with:

$ source stackrc
$ tripleo-ansible-inventory --list

In general, Ansible and the validations will be located on the undercloud, because it should have connectivity to all the overcloud nodes is already set up to SSH to them.

$ source ~/stackrc
$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory path/to/validation.yaml

Example: Verify Undercloud RAM requirements¶

The Undercloud has a requirement of 16GB RAM. Let’s write a validation that verifies this is indeed the case before deploying anything.

Let’s create validations/undercloud-ram.yaml and put some metadata in there:

---
- hosts: undercloud
  vars:
    metadata:
      name: Minimum RAM required on the undercloud
      description: >
        Make sure the undercloud has enough RAM.
      groups:
        - prep
        - pre-introspection

The hosts key will tell which server should the validation run on. The common values are undercloud, overcloud (i.e. all overcloud nodes), controller and compute (i.e. just the controller or the compute nodes).

The name and description metadata will show up in the API and the TripleO UI so make sure to put something meaningful there. The groups metadata applies a tag to the validation and allows to group them together in order to perform group operations, such are running them all in one call.

Now let’s add an Ansible task to test that it’s all set up properly. Add this under the same indentation as hosts and vars:

tasks:
- name: Test Output
  debug: msg="Hello World!"

When running it, it should output something like this:

$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory validations/undercloud-ram.yaml

PLAY [undercloud] *************************************************************

GATHERING FACTS ***************************************************************
ok: [localhost]

TASK: [Test Output] ***********************************************************
ok: [localhost] => {
    "msg": "Hello World!"
}

PLAY RECAP ********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

Writing the full validation code is quite easy in this case because Ansible has done all the hard work for us already. We can use the ansible_memtotal_mb fact to get the amount of RAM (in megabytes) the tested server currently has. For other useful values, run ansible -i /usr/bin/tripleo-ansible-inventory undercloud -m setup.

So, let’s replace the hello world task with a real one:

tasks:
- name: Verify the RAM requirements
  fail: msg="The RAM on the undercloud node is {{ ansible_memtotal_mb }} MB, the minimal recommended value is 16 GB."
  failed_when: "({{ ansible_memtotal_mb }}) < 16000"

Running this, we see:

TASK: [Verify the RAM requirements] *******************************************
failed: [localhost] => {"failed": true, "failed_when_result": true}
msg: The RAM on the undercloud node is 8778 MB, the minimal recommended value is 16 GB.

Because our Undercloud node really does not have enough RAM. Your mileage may vary.

Either way, the validation works and reports the lack of RAM properly!

failed_when is the real hero here: it evaluates an Ansible expression (e.g. does the node have more than 16 GB of RAM) and fails when it’s evaluated as true.

The fail line right above it lets us print a custom error in case of a failure. If the task succeeds (because we do have enough RAM), nothing will be printed out.

Now, we’re almost done, but there are a few things we can do to make this nicer on everybody.

First, let’s hoist the minimum RAM requirement into a variable. That way we’ll have one place where to change it if we need to and we’ll be able to test the validation better as well!

So, let’s call the variable minimum_ram_gb and set it to 16. Do this in the vars section:

vars:
  metadata:
    name: ...
    description: ...
    groups: ...
  minimum_ram_gb: 16

Make sure it’s on the same indentation level as metadata.

Then, update failed_when like this:

failed_when: "({{ ansible_memtotal_mb }}) < {{ minimum_ram_gb|int * 1024 }}"

And fail like so:

fail: msg="The RAM on the undercloud node is {{ ansible_memtotal_mb }} MB, the minimal recommended value is {{ minimum_ram_gb|int * 1024 }} MB."

And re-run it again to be sure it’s still working.

One benefit of using a variable instead of a hardcoded value is that we can now change the value without editing the yaml file!

Let’s do that to test both success and failure cases.

This should succeed but saying the RAM requirement is 1 GB:

ansible-playbook -i /usr/bin/tripleo-ansible-inventory validations/undercloud-ram.yaml -e minimum_ram_gb=1

And this should fail by requiring much more RAM than is necessary:

ansible-playbook -i /usr/bin/tripleo-ansible-inventory validations/undercloud-ram.yaml -e minimum_ram_gb=128

(the actual values may be different in your configuration – just make sure one is low enough and the other too high)

And that’s it! The validation is now finished and you can start using it in earnest.

For reference, here’s the full validation:

---
- hosts: undercloud
  vars:
    metadata:
      name: Minimum RAM required on the undercloud
      description: Make sure the undercloud has enough RAM.
      groups:
        - prep
        - pre-introspection
    minimum_ram_gb: 16
  tasks:
  - name: Verify the RAM requirements
    fail: msg="The RAM on the undercloud node is {{ ansible_memtotal_mb }} MB, the minimal recommended value is {{ minimum_ram_gb|int * 1024 }} MB."
    failed_when: "({{ ansible_memtotal_mb }}) < {{ minimum_ram_gb|int * 1024 }}"