Verification Component

Functional testing is a first step to ensuring that your product works as expected and API covers all use-cases. Rally Verification Component is all about this. It is not designed to generate a real big load (for this job we have Task Component), but it should be enough to check that your environment works by different tools (we call them Verification).

• Verifiers
  • What is it?
  • Verifier statuses
  • Verification statuses
  • Known verifier types
• Verification reports
  • HTML reports
  • Plugins Reference for all out-of-the-box reporters
• Command Line Interface
  • Category: verify
• HowTo
  • HowTo add new reporting mechanism
  • HowTo add support for new tool
  • HowTo migrate from Verification component 0.7.0 to 0.8.0

Historical background

Tempest, OpenStack’s official test suite, is a powerful tool for running a set of functional tests against an OpenStack cluster. Tempest automatically runs against every patch in every project of OpenStack, which lets us avoid merging changes that break functionality.

Unfortunately, it has limited opportunities to be used, to process its results, etc. That is why we started Verification Component initiative a long time ago (see a blog post for more details, but be careful as all user interface is changed completely since that time).

What is Verification Component and why do you need it?

The primary goal of Rally Product is to provide a simple way to do complex things. As for functional testing, Verification Component includes interfaces for:

• Managing things. Create an isolated virtual environment and install verification tool there? Yes, we can do it! Clone tool from Git repositories? Sure! Store several versions of one tool (you know, sometimes they are incompatible, with different required packages and so on)? Of course! In general, Verification Component allows to install, upgrade, reinstall, configure your tool. You should not care about zillion options anymore Rally will discover them via cloud UX and make the configuration file for you automatically.
• Launching verifiers. Launchers of specific tools don't always contain all required features, Rally team tries to fix this omission. Verification Component supports some of them like expected failures, a list of tests to skip, a list of tests to launch, re-running previous verification or just failed tests from it and so on. Btw, all verification runs arguments are stored in the database.
• Processing results. Rally DataBase stores all verifications and you can obtain unified (across different verifiers) results at any time. You can find a verification run summary there, run arguments which were used, error messages and etc. Comparison mechanism for several verifications is available too. Verification reports can be generated in several formats: HTML, JSON, JUnit-XML (see Verification reports for more details). Also, reports mechanism is expendable and you can write your own plugin for whatever system you want.