Using GBteSt
############

The main script is ``GBteSt.py``. Mandatory and optional command-line options are
provided. An help message can be printed using:

.. code-block:: shell

    > ./GBteSt.py --help
    usage: GBteSt.py [-h] [-r] [-c] --rootdir ROOTDIR [--rtol RTOL] [--atol ATOL] [-o OUTPUT_FORMAT] [-n OUTPUT_NAME]

    options:
      -h, --help            show this help message and exit
      -r, --run             Run test scripts
      -c, --compare         Compare test results
      --rootdir ROOTDIR     Root folder where to store all the test results
      --rtol RTOL           Tolerance for comparing relative differences with floats
      --atol ATOL           Tolerance for comparing absolute differences with floats
      -o OUTPUT_FORMAT, --output_format OUTPUT_FORMAT
                            Format for the comparison output (text or html)
      -n OUTPUT_NAME, --output_name OUTPUT_NAME
                            Name of the output file (to be stored in --rootdir)

The different options are:

 - ``--run``: run the tests defined by the configuration files
 - ``--compare``: compare the results obtained via the ``--run`` option. Each
   dataset of the result files are compared using: 
   
   :math:`|result\_1 - result\_2| < a_{tol}+ r_{tol} \cdot |result\_1|`,
   
   where ``atol`` and ``rtol`` are defined below. Setting ``atol > 0`` and ``rtol = 0``
   compares the results using and absolute difference, while ``atol = 0`` and
   ``rtol > 0`` compares them using a relative difference.
 - ``--rootdir``: root directory where to store the test results and the summary file.
 - ``--rtol``: tolerance for the relative difference.
 - ``--atol``: tolerance for the absolute difference.
 - ``--output_format``: format in which the summary is generated. Options are
   "html" or "terminal".
 - ``--output_name``: name of the file containing the summary of the test.

Testing workflow
================

 1. First, you need to setup your tests according to the [configuration
    page](configuration.md). Note that the executables must be already compiled.
 2. Once the test setup is configured, you can run the tests using:

    .. code-block:: shell

       ./GBteSt.py --rootdir <rootdir> --run

    This will launch all the test parametrized in the configuration file.
 3. When the tests have all run, you can compare the results and generate a
    summary report using:

    .. code-block:: shell

       ./GBteSt.py --rootdir <rootdir> --compare --rtol <rtol> --atol <atol> -o ''

    By default, this will print the summary to the terminal. If you wish to
    generate an HTML report, add the ``-o`` and ``-n`` options.