Compliance benchmarking is done via the OGC Compliance & Interoperability Testing & Evaluation Initiative. The pycsw wiki documents testing procedures and status.
The pycsw tests framework (in tests) is a collection of testsuites to perform automated regession testing of the codebase. Test are run against all pushes to the GitHub repository via Travis CI.
The tests framework can be run from tests using Paver (see pavement.py) tasks for convenience:
$ cd /path/to/pycsw
# run all tests (starts up http://localhost:8000)
$ paver test
# run tests only against specific testsuites
$ paver test -s apiso,fgdc
# run all tests, including harvesting (this is turned off by default given the volatility of remote services/data testing)
$ paver test -r
The tests perform HTTP GET and POST requests against http://localhost:8000. The expected output for each test can be found in expected. Results are categorized as passed, failed, or initialized. A summary of results is output at the end of the run.
If a given test has failed, the output is saved in results. The resulting failure can be analyzed by running diff tests/expected/name_of_test.xml tests/results/name_of_test.xml to find variances. The Paver task returns a status code which indicates the number of tests which have failed (i.e. echo $?).
The tests framework is run against a series of ‘suites’ (in tests/suites), each of which specifies a given configuration to test various functionality of the codebase. Each suite is structured as follows:
When the tests are invoked, the following operations are run:
The CSV format of tests/suites/suite/get/requests.txt is testname,request, with one line for each test. The testname value is a unique test name (this value sets the name of the output file in the test results). The request value is the HTTP GET request. The PYCSW_SERVER token is replaced at runtime with the URL to the pycsw install.
To add tests to an existing suite:
To add a new test suite:
The new test suite database will be created automatically and used as part of tests.
You can also use the pycsw tests via your web browser to perform sample requests against your pycsw install. The tests are is located in tests/. To generate the HTML page:
$ paver gen_tests_html
Then navigate to http://host/path/to/pycsw/tests/index.html.