Unable to run models in CPU for inference

## ❓ Questions and Help
Hi I would like to run some checkpointed models in CPU , unable to do so. Inference is still on CUDA
In `mmf/configs/defaults.yaml`, I set
```
training:
      device: cpu
evaluation:
      use_cpu: true
```