Two different types of integration tests are commonly referred to as black box testing and white box testing.
White box testing refers to tests where the test runner can access internal data structures (machine state) or algorithms in order to verify a specification.
In black box testing, only the public API or other externally facing resources are ever called, since internal interfaces would give access to internal machine state, and the goal is typically to verify that a public API behaves according to its publically documented specification.
There was a proliferation of integration test suites in 2011, and at the Essex design summit, a number of interested parties, including authors of the various existing integration test suites – met to discuss a plan to consolidate similar test suites into a single project that everyone could focus their efforts on. The resulting project is the Tempest project. This project contains functional and integration tests that are meant to be executed against a fully-deployed OpenStack environment.
The project source code is divided up in such a way that additional or alternate functionality can easily be added to the framework. The current directory structure is shown below.
Of these, the include, kong, and tools directories are deprecated and will soon be removed. Of the remaining directories, their purposes are:
When designing this framework, the primary focus were the following goals: ease of development, scalability, flexibility, and maintainability. This means that a test class can take action via the client without having to format and make a request. The client then becomes a middle man between the tests and a simple REST client. This separation of concerns has the benefit of not only x, but also reduces the refactoring impact when changes are made to the application under test.
To achieve the first goal, domain specific clients were developed. In doing so, several things were achieved. First, some of the complexity of testing APIs such as handling requests, authentication, and parsing of responses are already handled. An additional benefit is that in doing so, the structure and logic of the REST request itself is removed from the tests into a single location. This increases the maintainability of the code since when a change in logic or request structure occurs, only the client code needs to change.
To make execution of Tempest flexible, values that will vary by environment are expected to be provided by an external data source, which is currently an external configuration file.
Additionally, Tempest takes into account that certain functionality may vary or not be usable based on the environment the tests are executed against. To avoid execution of tests that are not valid in a given environment, feature flags can also be set which will skip over executing tests that are not valid for the environment under test. For more detailed information, see the Temptest_README file and the example Tempest configuration file which are located in the /etc directory.
Tests can be run using the Nose test runner. To run a test or a group of tests, execute the nosetests command with a specific test file or directory as a parameter. For example, running “nosetests tempest/tests” will run all tests in the tempest/tests directory.
In the development of tests, some patterns and practices have come into being: