doctests

Introspective micro-atomic-unittests by means of docstrings

Test Driven Development is a widely used software developing model ensuring many good* features such as validation, clean up, reduced debugging effort (regression tests), etc... All of these aspects increase the overall code quality. However, writing tests could be time consuming especially for large libraries relying on many procedures: often, it can be helpful to have the possibility to test micro (atomic) snippets of code without the necessity to write a real complex fully-featured test.

FoBiS allows (with some limitations, see below) to write a very concise micro test directly inside the doc-strings commenting your codes (alleviating from the necessity to write a fully-featured test) and to automatically execute your tests checking the results against the ones you expect!

This feature adds a sort of introspective capability to your codes as it is common in the framework of interpreted language like Python. Indeed, this feature of FoBiS is inspired by the doctest module of Python.

How it work?

As the doctest Python module, FoBiS searches for snippets of codes inside your commenting doc-strings that look like interactive test session, the syntax of which is described below. Once a doctest is found a volatile test program is created, compiled and automatically executed for you and the result obtained is compared to the result that you specify as expected. The volatile test program sources and executable is automatically removed after the execution, but by means of a specific CLI flag they could be kept for debugging purposes.

doctest syntax

You can place a doctest anywhere into your sources. FoBiS creates a hierarchy of doctests for each module parsed accordingly a sequential numerical ordering. Let us consider a simple example.

module simple
contains
  function add(a, b) result(c)
  !< Add two integers.
  integer, intent(IN) :: a
  integer, intent(IN) :: b
  integer             :: c
  c = a + b
  endfunction add

  function sub(a, b) result(c)
  !< Subtract two integers.
  integer, intent(IN) :: a
  integer, intent(IN) :: b
  integer             :: c
  c = a - b
  endfunction sub
endmodule simple

It could be helpful to test atomically this kind of module procedures without the necessity to write a real test program, to compile the test, to execute the test and to check visually the results. All these steps can be automatically accomplished by means of FoBiS doctests. The above example equipped with such a magic wand looks like

module simple
contains
  function add(a, b) result(c)
  !< Add two integers.
  !<```fortran
  !< print*, add(a=12, b=33)
  !<```
  !=> 40 <<<
  !< @note This test is wrong intentionally...
  integer, intent(IN) :: a
  integer, intent(IN) :: b
  integer             :: c
  c = a + b
  endfunction add

  function sub(a, b) result(c)
  !< Subtract two integers.
  !<```fortran
  !< print*, sub(a=12, b=33)
  !<```
  !=> -21 <<<
  integer, intent(IN) :: a
  integer, intent(IN) :: b
  integer             :: c
  c = a - b
  endfunction sub
endmodule simple

We have add only 4 lines of comments for each procedure!

The syntax is based on the same syntax used by FORD documenting tool for including code snippet into the documentation. In particular the first 3 lines will be rendered by FORD as a (colored) code snippet, while the last line will be ignored, it not being a valid docstring of FORD.

The syntax is the following:

!$```fortran
!$ #your_test
!$```
!=> #expected_result  <<<

where the $ character can be replaced with any characters you prefer (trick: use the one adopted for FORD documentation generation). The body of the test must be enclosed into a pair

doctest start

!$```fortran

doctest end

!$```

while the expected result must follow the test body and must be enclosed into the pair

doctest result

!=> #expected_result  <<<

The body of the doctest can contain any valid Fortran codes, including definition of variables, use statements, definition of types, etc... For example the following are all valid doctests

module less_simple
contains
  function add(a, b) result(c)
  !< Add two integers.
  !<```fortran
  !< type :: foo
  !<   integer :: a(2)
  !< endtype foo
  !< type(foo) :: bar
  !< bar%a = 1
  !< print*, add(a=bar%a(2), b=bar%a(1))
  !<```
  !=> 2 <<<
  integer, intent(IN) :: a
  integer, intent(IN) :: b
  integer             :: c
  c = a + b
  endfunction add

  function sub(a, b) result(c)
  !< Subtract two integers.
  !<```fortran
  !< integer :: a, b
  !< a = 2
  !< b = 4 * a
  !< a = add(a, b)
  !< print*, sub(a, b)
  !<```
  !=> 2 <<<
  integer, intent(IN) :: a
  integer, intent(IN) :: b
  integer             :: c
  c = a - b
  endfunction sub

  subroutine multiply(a, b, c)
  !< Multiply two integers.
  !<
  !<### Introspective doctests
  !<```fortran
  !< integer :: c
  !< call multiply(a=3, b=4, c=c)
  !< print*, c
  !<```
  !=> 12 <<<
  !<
  !<```fortran
  !< integer :: c
  !< call multiply(a=-2, b=16, c=c)
  !< print*, c
  !<```
  !=> -32 <<<
  integer, intent(IN)  :: a
  integer, intent(IN)  :: b
  integer, intent(OUT) :: c
  c = a * b
  endsubroutine multiply
endmodule less_simple

It is natural to place doctests inside the documentation of each procedure, but this is not a prescription, you can place theme everywhere. For example, it could be helpful to have module-level doctests

module simple
contains
  !< Simple module.
  !<### Regression tests and usage example
  !<##### Add
  !<```fortran
  !< print*, add(a=12, b=33)
  !<```
  !=> 45 <<<
  !<##### Subtract
  !<```fortran
  !< print*, sub(a=12, b=33)
  !<```
  !=> -21 <<<

  function add(a, b) result(c)
  !< Add two integers.
  integer, intent(IN) :: a
  integer, intent(IN) :: b
  integer             :: c
  c = a + b
  endfunction add

  function sub(a, b) result(c)
  !< Subtract two integers.
  integer, intent(IN) :: a
  integer, intent(IN) :: b
  integer             :: c
  c = a - b
  endfunction sub
endmodule simple

Note that using the same syntax for both FoBiS doctests and FORD code snippets the doctests can serve twofold purposes: they are atomic-regression tests and in the meanwhile they are examples of usage!

executing doctest

Once you have equipped your code with valid doctests their execution and validation are very simple, just run FoBiS in doctests mode:

FoBiS.py doctests

For example, running FoBiS on the simple example above we will obtain something like:

executing doctest simple-doctest-2
doctest passed
executing doctest simple-doctest-1
doctest failed!
  result obtained: "45"
  result expected: "40"

The output is not ordered.

To the doctests FoBiS.py mode can be passed any valid build options and also all the features associated to the fobos file. Moreover, doctests mode has one specific CLI flag:

FoBiS.py doctests -keep_volatile_doctests

Passing this switch the volatile test programs that FoBiS automatically creates/compiles/executes are not removed after the tests check.

The volatile programs are saved into the build directory (specified by means of CLI of fobos option): for each module source containing doctests a subdirectory named as the module is created into the build root directory and inside each module subdirectory a Fortran program source is created for each doctest. The doctests are then compiled and executed and their output is captured by FoBiS and compared with the expected one. For example, running FoBiS on the simple example above we will create a structure of files like the following

FoBiS.py doctests --build_dir build
tree
.
├── build
│   ├── doctests-src
│   │   └── simple.f90
│   │       ├── simple-doctest-1.f90
│   │       ├── simple-doctest-1.result
│   │       ├── simple-doctest-2.f90
│   │       ├── simple-doctest-2.result
│   ├── simple-doctest-1
│   ├── simple-doctest-2
│   ├── mod
│   │   └── simple.mod
│   └── obj
│       ├── simple-doctest-1.o
│       ├── simple-doctest-2.o
│       └── simple.o
└── src
    └── simple.f90

Limitations

Unfortunately, Fortran has not introspective capabilities and this poses some limitations on what FoBiS can (easily) do. Presently, there are two main limitations:

1. the result must be printed to the standard output inside the doctests body: as a matter of facts, FoBiS can capture the doctests results only by means of the standard output;
2. only public objects contained into modules can be tested: the private objects (types, variables and procedures) that are not public cannot be doc-tested by FoBiS.

While the first limitation is not so hurting (at least for me), the second generates some frustrations: I hope to find a (easy) way to relax this limiation.