Callbacks

Callbacks are the main abstraction powering the training loop, so it is important to get them right.

Conflicts with callbacks

The callback system in fastai is very powerful but also not very safe, since conflicts between callbacks can arise when state is being updated.

There are no conflicts as long as all state accesses are read-only. Conflicts appear when state is updated or created.

• When state is created, dependencies must run after the creation.

  Example: Recorder creates a history and ProgressCallback needs to access that history

• When state is updated, the ordering of dependencies is not clear.

  Should state reads always be put after updates/writes?

  If two callbacks update the same state at the same phase, which should run first?

Solutions

A trade-off between power and safety needs to be made. The callback system in fastai is very powerful, but it does almost nothing to prevent invalid state and the conflicts that can arise.

Here are some possible approaches to mitigate, prevent and find problems:

1. Make callbacks have read-only access to state by default. This makes them harder to misuse.

   Callbacks can implement an interface that defines what state they are allowed to update.

2. Callbacks specify their dependencies on other callbacks. This allows the construction of a directed acyclic graph that can be used to execute the callbacks in a valid order.

3. Find possible conflicts and require them to be resolved. If two callbacks both define that they need to update the same state at the same time during training, there should be some resolution method that defines either a) which one goes first, b) that the order doesn't matter or c) that both callbacks cannot work together at all.

Requirements

TBD.

---

For gathering requirements on how callbacks are being used, here is a survey of their usage in fastai: Callbacks and State in fastai. Based on this it is possible to create a list of possible conflicts and see where conflicts actually happen.