Skip to content

Latest commit

 

History

History
122 lines (91 loc) · 3.9 KB

readme.md

File metadata and controls

122 lines (91 loc) · 3.9 KB
Raw

Model-Based Active Exploration (MAX)

Code for reproducing experiments in Model-Based Active Exploration, ICML 2019

Written in PyTorch v1.0.

Code relies on sacred for managing experiments and hyper-parameters.

Overview:

  • envs/: contains the environments used.
  • main.py: contains the main algorithm and baselines through modes.
  • models.py: a fast parallel implementation of an ensemble of models which can are trained with negative log-likelihood loss.
  • utilities.py: contains the all the utilities (exploration objectives) used in the paper.
  • imagination.py: contains code that constructs a virtual MDP using the model ensemble.
  • sac.py: contains a simple Soft Actor-Critic implementation.
  • sacred_fetcher.py: script to download experiment artifacts stored in MongoDB.

Installation

  • Install required dependencies:

    sudo apt install libosmesa6-dev libgl1-mesa-glx libglfw3 patchelf

  • Create conda environment with required dependencies:

    conda env create -f conda_env.yml

  • Download and setup MuJoCo binaries. The project uses mujoco and mujoco_py version 1.50.

    mkdir ~/.mujoco/
cd .mujoco/
wget -c https://www.roboti.us/download/mjpro150_linux.zip
unzip mjpro150_linux.zip
rm mjpro150_linux.zip

    Obtain MuJoCo license key and place it .mujoco/ directory created above with filename mjkey.txt.

  • Append the following to ~/.bashrc:

    # MuJoCo
export LD_LIBRARY_PATH=:/home/<USER>/.mujoco/mjpro150/bin

if [ -f /usr/lib/x86_64-linux-gnu/libGLEW.so ]; then    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/<USER>/.mujoco/mjpro150/bin:/usr/lib/nvidia-390
    export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia-375
fi

  • Quick test of MuJoCo installation

    >>> import gym
>>> gym.make('HalfCheetah-v2')

Commands

Execute the commands listed below from the code directory to reproduce the results.

Half Cheetah

  • MAX:
python main.py with max_explore env_noise_stdev=0.02
  • Trajectory Variance Active Exploration:
python main.py with max_explore utility_measure=traj_stdev policy_explore_alpha=0.2 env_noise_stdev=0.02
  • Renyi Divergence Reactive Exploration:
python main.py with max_explore exploration_mode=reactive env_noise_stdev=0.02
  • Prediction Error Reactive Exploration:
python main.py with max_explore exploration_mode=reactive utility_measure=pred_err policy_explore_alpha=0.2 env_noise_stdev=0.02
  • Random Exploration:
python main.py with random_explore env_noise_stdev=0.02

Ant

  • MAX:
python main.py with max_explore env_name=MagellanAnt-v2 env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True
  • Trajectory Variance Active Exploration:
python main.py with max_explore env_name=MagellanAnt-v2 utility_measure=traj_stdev policy_explore_alpha=0.2 env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True
  • Renyi Divergence Reactive Exploration:
python main.py with max_explore env_name=MagellanAnt-v2 exploration_mode=reactive env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True
  • Prediction Error Reactive Exploration:
python main.py with max_explore env_name=MagellanAnt-v2 exploration_mode=reactive utility_measure=pred_err policy_explore_alpha=0.2 env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True
  • Random Exploration:
python main.py with random_explore env_name=MagellanAnt-v2 env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True

Magellan

Magellan is the internal code name of the project inspired by life of Ferdinand Magellan.