Moving forward there is a desire to reduce the scope of the gate to a stable and reliable "gold standard" and move the testing of special cloud configurations and scenarios to post merge. [0] This is to help reduce the complexity of the gate to make it more reliable and easier to debug for everyone. But, first we need to define what kind of scenarios are outside of the gold standard and should be tested post merge. There is also a gap in the tooling for this effort, we need a dashboard and other tools to advertise the test results properly and to simplify troubleshooting.
Discussion on test after code merge:
- Which features to verify after merge
- Which kind of tests (flacky, long running, multinode)
- How to track results trends
- How to save features and tests verified after merge from bit rot
[0]
http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html
