Skip to content

Home

Jump to bottom
Kevin Egger edited this page Apr 12, 2018 · 8 revisions

The purpose of this repository is to provide a generic framework for evaluation.

Building Blocks (outdated)

Experiments

Experiments are the basic building blocks of the evaluation framework. Each experiment is defined in a YAML file and stored in the experiments folder, see here: https://github.com/ethz-asl/evaluation_tools/blob/master/experiments/sample_experiment.yaml

An experiment defines:

  • the executable to be run,
  • the dataset(s) to use,
  • the calibration file(s),
  • the runtime parameters,
  • the (optional) pre-processing scripts to be run,
  • the post-processing scripts to be run.

To run the experiments, use the run_experiments.py script.

Datasets

Experiments must specify which dataset or datasets will be used, by listing them in the datasets tag of the experiment.yaml file. For now, only rosbags are supported.

The script will look for the rosbags in two places:

  • First, it will try to find them inside your evaluation_tools/datasets folder.
  • In case they are not there, it will look at the entries of the datasets.yaml file. If there is a match, it will try to download the dataset into your local folder and use it from there.

Datasets for evaluation are stored on the mapbox for now, under /mapbox-public/Datasets.

Whenever you want to use a dataset that is not listed in the datasets.yaml file but is online, please expand the list with the name and path of the dataset inside the NAS. This way, other machines will be able to run your experiments.

Calibration

Calibration files should be checked in here in the calibration folder, and structured by robot/sensor-name, see here: https://github.com/ethz-asl/evaluation_tools/blob/master/calibrations/VI-Sensor/23011/ncamera_MVI1_0416.yaml

Parameters

The parameter_files folder contains parameters in the form of YAML files. Example: swe_test_parameters.yaml

There are four strings that act as placeholders for information that, in spite of being runtime parameters, should not be defined in these parameter files:

  • LOG_DIR. This is the path to the folder where the results are stored, and can be specified via command-line,
  • NCAM_CALIB_FILENAME. This is the path to the camera calibration file, and is specified in the experiment yaml file,
  • ODOM_CALIB_FILENAME. Same as above, but for the odometry calibration file,
  • BAG_FILENAME. This is also specified in the experiment yaml file.

run_experiments.py

https://github.com/ethz-asl/evaluation_tools/blob/master/python/run_experiment.py This is the hook to get started.

How it works (outdated)

In order to get an intuition of how the run_experiments.py script works, please take a look at this diagram, which includes a flowchart that explains how the script runs, as well as some sample configuration files (you can modify the diagram and copy parts of it if you click in the 'Open in New Window' button on the bottom).

Clone this wiki locally