Declarative test framework¶

Problem description¶

Currently, running tests in Arvados means running the monolithic run-tests.sh script. The current script has several problems:

• Script is long, complex and hard to maintain
• Script sets up its own environment to run tests is substantially different from production environment
• Setting up the test environment adds overhead, even when there is nothing to do
• Definition of setting up and executing tests is separate from the directory containing the component under test.
• Related tasks such as building and testing packages are entirely separate.

Solution¶

I think we want a declarative way to describe our components for the purposes of building, testing and packaging. This would provide the metadata necessary to drive a framework which is able to build, test and package each component based on that metadata.

This will make it possible to drive framework improvements independently of the individual test definitions, such as automatically generating jenkins pipeline XML to run the tests.