An implementation of Cognitive Mapping and Planning for Visual Navigation (S. Gupta et al, CVPR 2017) neural network architecture.
Be warned. I did not succeed in replicating results in a continuous environment (such as DeepMind Lab). Regardless, if you would like to take a stab at this model, the multiprocdmlab package is available here. I cannot provide support on this repository.
The paper introduced an interesting architecture where an explicit planning module using Value Iteration Networks are used. The network consists of two major modules:
- Mapper, which takes a first person visual input and maps it to a top-down 2D free space estimate.
- Planner, which takes the output from the 2D free space estimate, runs value iteration and finally outputs an action.
More a more detailed implementation please refer to the paper.
model.py should be self contained. Other environment training files (such as environment.py or expert.py for expert data generation) are not expected to work without some modifications to the DeepMind Lab release.
While the original architecture is designed to solve a navigation problem in a simulated in-door environment, this architecture is designed for testing in DeepMind's Lab navigation environment. As such, several modifications were made.
- The mapper architecture also keeps a copy of "rewards map" -- unlike the paper environment where the goal is clearly defined, the agent in Lab environment must explore first.
- The ResNet autoencoder has been replaced with a simple two-layer autoencoder.
- Sampling and VIN modules are implemented using RNN cells.