Running an experiment

To run an experiment, run the following command:

rosrun evaluation_tools run_experiment.py <path_to_experiment_yaml>

Additional flags:

• results_output_folder: where to run the experiment and where to store its results.
• automatic_download: if specified, unavailable datasets will be downloaded without asking the user.

Job structure

For each (dataset, parameter) pair, a job folder will be created inside the results folder with the name <timestamp>__<experiment_name>. In this folder, a folder <dataset_name>__<parameter_name>, the job folder, will be created for each pair. For multi-map experiments, a folder is only created for each parameter entry (there will be one job for all datasets).

The job folder contains the following files:

console_commands.yaml

File containing all console commands to be run. This is the input file for the console batch runner.

estimator_output_<dataset_name>

Output folder for user-defined estimator-specific data.

In multi-map experiments, one folder is created for each dataset.

job.yaml

Contains all the relevant information to run a job.

job_summary.yaml

Unknown.

batch_runner.INFO/WARNING/ERROR

glog output of the console batch runner

User-defined files

The experiment may write additional files to disk from the estimator (as defined in the parameter file), the console or the evaluation script. These should be saved somewhere inside the job folder.