README

IPET (Interactive Performance Evaluation Tools) is a toolbox that
allows to easily create customized benchmark tables from
raw solver log files. It is aimed to develop evaluations through a
grapical interface that can later be loaded and reused on other
data. Additional readers and evaluations are stored in
xml-format which can be plugged in into the evaluation method.

Some visualizations are also provided.

Installation and prerequisites
------------------------------

This section covers the installation of IPET. We assume that you have a working
Python 2.7 installation. On top of that, we require
 - matplotlib version 1.3.1 or higher
 - Tkinter with Tcl/Tk version 8.5 or higher
 - pandas version 0.15.0 or higher
 - for some distributions, the Pillow library as a replacement for the PIL library.

In order to install the IPET-module and its contents to the python path and
thus make it importable from every working directory you are in,
run the command:

 python setup.py install [--user]

This will install IPET to your Python distribution, and make
it importable from every working directory in the file system. If
you do not have administrator rights on your system, append '--user'
in order to install the package to your local Python libraries.
If you now start an interactive python interpreter from any working directory,
you can import IPET as a module into your Python scripts and use its methods.

The user interface uses the Python Image Library (PIL) to display button images
via PNG files. In case you have an error that says PIL could not be found, install
Pillow via

 pip install Pillow [--user]

or

 python easy_install Pillow [--user]



Overview of the library
-----------------------

The IPET library consists of two main parts: a back end, which is responsible
for data acquisition, data storage, and file IO, and a front end, which
consists of the necessary components to interact with the back end data
through a user interface.
The central back end controller class is called
TestRun, and resides in a module of the same name.
In order to acquire solver benchmark data, we have to feed our program
with solver output in the form of log files.
It is safe to rely on the format of ".out"-files that are generated by SCIP
when running the
    make test
command.
Every log file is represented by an instance of the TestRun class.

An object
of the Comparator class comprises an arbitrary number of TestRun
instances, which are associated to a (list of) SCIP log files. Data are
read by means of StatisticReader instances. A newly constructed
Comparator instance comes with a large variety of available readers for
parsing the solving time and number of branch-and-bound nodes, but also
more advanced readers that read in complete tables, or so-called histories;
the latter combine column output from the periodic SCIP status table, such
as the development of the primal bound as a function of the solving time. It is also
possible to add additional readers and reevaluate the log file information.

After the collection of the data has been finished, the
acquired data for each TestRun is stored as DataFrame object
of the pandas library.
Data frames allow for fast data manipulations such as grouping data, transformations,
aggregations, and combinations of them. The data can also be exported to several
formats such as, e.g., LaTeX, Microsoft Excel ".xls"-format, or ".csv"-files.


How to use ipet
-------------------
1) parse the .out files with "python parsetestruns.py [outfile]" or "python parsetestruns.py (outfile 1) (outfile 2) ... (outfile n)" to .trn files.

2) compare some .trn files with " python evaluate.py -e (evaluation file) -t (.trn file 1) (.trn file 2) ... (.trn file n) "