During functional testing the program is tested as a whole. This is much like if the testing were made by a human. Although the user interface parts are not tested the core of the program is tested and how the different program components work together. When functional tests succeed that means that the tested scenarios will most likely work properly when the program is run by real users. The quality of total testing obviously depends on the amount and quality of individual tests.
Note: In addition to functional tests there are unit tests, which are not covered by this document.
The functional testing in NZBGet is controlled by scripts, which do things like:
NZBGet downloads from Usenet servers. However it would be unreliable and slow to use global Usenet servers for functional testing. Instead we need to do all the testing locally using our set of test files. For this purpose a simple NNTP server called NServ was developed.
Technically NServ is integrated into NZBGet as a sub-module. NServ is automatically controlled (started, stopped) by functional testing scripts but it can also be used standalone for other purposes such as debugging NZBGet or for speed tests of NZBGet or even other download clients. For more info see NServ NNTP server.
Functional tests for NZBGet are written in python language. We are using testing framework py.test to organize test execution and collect test results. Therefore in order to run functional tests py.test must be installed on your system.
Functional tests and supporting modules are stored in directory
tests/functional. To run tests open command prompt in this directory and execute
py.test as explained below.
Tests can be run without extra configuration in which case the test files are stored in
tests/testdata/nserv.temp and temporary files created by NZBGet during testing are stored in
tests/testdata/nzbget.temp. Both of these directories take several gigabytes of disk space and can be reconfigured to use other locations.
Create text file
tests/functional/pytest.ini with the following content:
[pytest] nzbget_bin=/path/to/nzbget sevenzip_bin=/path/to/7za nserv_datadir=/path/to/nserv.temp nzbget_maindir=/path/to/nzbget.temp
All settings in the ini-file are optional.
When executing the tests for the first time the test scripts prepare test files and put them into directory
tests/testdata/nserv.temp. These files take several gigabytes and may need several minutes to generate. When starting tests for the first time it’s recommended to add parameter
-s to see additional progress logging during preparation stage:
py.test -v -s
To run all tests use simple command:
To run tests from one test script only pass the specific file name:
py.test -v parcheck\parcheck_force_test.py
To execute only one specific test pass the file name and test function name:
py.test -v parcheck/parcheck_auto_test.py::test_parchecker_repair
For more filter possibilities please see py.test documentation.
Directory used by NZBGet during testing (
tests/testdata/nzbget.temp by default) is automatically deleted if all tests succeed. If a test fails the directory is kept in order to preserve log-files and queue-files for failure analysis. However during testing NZBGet is started and stopped multiple times. If the failed test wasn’t the last test the preserved NZBGet directory may not contain data of the failed test. In such case it’s better to rerun the specific failed test as the only test (using specific command line).
After test completion NZBGet is terminated. To analyze a test failure it can be useful to have NZBGet running, for example to inspect the state in web-interface. Pass extra parameter
--hold to achieve this:
py.test -v --hold parcheck/parcheck_auto_test.py::test_parchecker_repair
--hold in combination with one test only.