Rally v0.4.0¶
Information¶
| Commits | 76 |
| Bug fixes | 12 |
| Dev cycle | 28 days |
| Release date | 4/18/2016 |
Details¶
Warning
Rally DB schema was changed since previous release. See HOWTO about updating your database.
CLI changes¶
- Add status messages of db migration process
- Display task errors in human-friendly form
- Support OS_PROJECT_NAME as well as OS_TENANT_NAME
Messages¶
Removed deprecation warning in case of transmitted "name" attribute while creation neutron resources.
Warning
Deprecated code was deleted.
Suppress warning insecure URL messages
Do not spam end users by insecure URL messages because it is quite valid case in testing process
Database¶
While preparing for deployment refactoring:
- db schema was changed;
- migration with new column credentials to deployment model was added;
- columns users and admin were dropped.
Rally Task¶
Remove deprecated scenario output mechanism via returing value
Warning
Deprecated code was deleted.
Friendlier error message with empty task file
This is particularly useful when a Jinja2 template results in an empty task. The current error message isn't very helpful:
Task config is invalid: 'NoneType' object has no attribute 'get'
Add Heat template validator
Plugins¶
Scenarios:
- Extend VM bind actions with "pause_unpause", "suspend_resume", "lock_unlock", "shelve_unshelve".
- Add exact error message into VMTasks.runcommand_heat scenario
Add heat scenarios: output-show, output-list
- Current patch contains 4 scenarios from heat repo:
Contexts:
- Reduce default speed of users creation in users context from 30 to 20 by default.
SLAs:
NEW!! MaxAverageDurationPerAtomic : Maximum average duration of one iterations atomic actions in seconds.
Reports:
- Improve results calculation in charts.Table
- Use int instead of float for Y axis. It's number of parallel iterations and it can't be float.
- Remove accuracy that makes no sense, and creates a lot of noise on this graph
- Include failed iterations as well, otherwise we will calculate load incorrectly
- Graph should start from 0 (begging of experiment)
- Add 2 points at the end of graph to get at the end of graph 0 iterations in parallel
Task Exporter:
In previous release we introduced new mechanism to export results in various external systems and various formats.
In this release, we added first plugin for this stuff - file_exporter
Services:
Remove hardcoded timeout from heat service
Utils:
Make glance web uploads streamable
Without this change entire file get's downloaded into memory and can cause issues.
Rally Verify¶
Set time precision to 3 digits (instead of 5) after dot.
Don't use "--parallel" flag when concurrency == 1
If concurrency equals to 1, it means that we use only one thread to run Tempest tests and the "--parallel" flag is not needed.
Plugin for DevStack¶
Support to be enabled with different plugin name
Allow rally to be installed by devstack through a different plugin name, e.g:
enable_plugin test-rally http://github.com/rally/rally.git master
Removed uncalled code
Devstack won't "source plugin.sh source" any more.
Bug fixes¶
12 bugs were fixed:
X-Fail mechanism did not work for TestCase which failed on setUp step
If Tempest fails in a test's setUpClass(), there is only one subunit event for each TestCase. In this case, Rally did not check partial test with x-fail list and marked test as "fail" insted of "x-fail".
Weak isolation of scenario arguments between iterations
Input arguments for sub-task were shared between all iterations. Rally team found one scenario which modified mutable input variable.
Affected scenario: NeutronNetworks.create_and_update_ports
Incompatible filters between V1 and V2 for Glance images listing
Glance V1 and V2 have different filters. For example, "owner" is a separate kwarg in V1, not a generic filter. Also, visibility has different labels in different APIs. We modified our Glance wrapper to support Glance V2 format of filters for both V1 and V2
Wrong way to store validation errors
Results of failed task validations saved in incorrect format. It broke and made un-userfriendly rally task detailed command.
Hardcoded task's status in rally task results
If there are no results for task, rally task results printed message that task has failed status, but it can be not true(tasks in running state do not have results).
Tempest context failed to create network resources
While we merged improvement for keystoneclient, we used wrong way to obtain tenant id in TempestContext.
Tasks based on Tempest failed to parse execution time.
There is an ability in Rally to launch tasks based on Tempest. Since launch of Tempest is just subprocess, it is needed to parse subunit to set correct atomic actions.
There was an issue while converting task execution time.
JSONSchema huge impact on task performance
Before runner sent data to engine we were checking jsonschema. This operation is very expensive and in some cases it can take a lot of time.
Here are test results, with Dummy.dummy_output scenario, sleep 0.5s (added manually), 8000 iterations, 400 in parallel:
- on master branch before the fix:
- Load duration: 117.659588099 Full duration: 227.451056004
- on master before the fix but remove jsonschema validation in scenario:
- Load duration: 12.5437350273 Full duration: 128.942219973
- on this patch before the fix (pure python validation):
- Load duration: 11.5991640091 Full duration: 22.7199981213
Wrong Calculation of running iterations in parallel
Load profile chart was calculated wrongly. It showed more running iterations in parallel than actually are running.
Rally did not show "missing argument" error raised by argparse while parsing cli args
Issue while checking required arguments in CLI
There was a possible issue in case of several required arguments
Prepare step of verification did not check visibility of obtained image
When we request a list of images to choose one of them for tests, we should make sure all images are active and they are PUBLIC. If images are not public, we will have failures of Tempest tests as described in the bug.
Thanks¶
2 Everybody!