Best way to evaluate downstream performance?

[astrobdr]

Suppose we are training a resnet using a scheme like moco.

And between every epoch (or every few epochs) we want to freeze the weights and evaluate the trained feature extractor on a different dataset by just attaching a 2-layer MLP to the end of the frozen feature extractor, training the MLP for a few epochs.

Whats the best place to do this downstream training? In a callback? Is there already an example available?

Some caveats: I would have to make new dataloaders in this 2nd-stage training, with a different dataset, and start with a random MLP every time.

Advice?

[rohitgr7]

hey @smartdanny !

can you explain your tasks a little more? like step-wise procedure on how it should work.
Do you want to evaluate with new MLP layers, or train it too?
Do you want to do all of this after N epochs or after every N epochs?
Do you want to unfreeze the feat-extractor at any time?

[astrobdr]

can you explain your tasks a little more? like step-wise procedure on how it should work.

1. Train a feature extractor (lets say resnet) for 5 epochs.
2. Freeze above feature extractor.
3. replace its head with 2 MLP layers.
4. Train MLP layers on different dataset
5. Evaluate performance of "frozen feat-extractor" + MLP on this external dataset (that the feature extractor never trained on)
6. Done

Do you want to evaluate with new MLP layers, or train it too?

• Yes also train. Once the feature extractor is trained, I need to freeze it, then train some MLP layers using those extracted features, and then evaluate the output of the MLP layers.

Do you want to do all of this after N epochs or after every N epochs?

• Doesnt matter. We can just assume I want to evaluate the "downstream" performance of the feature extractor every 5 epochs. AKA, train feature extractor for 5 epochs, replace its head with an MLP, train only the MLP layers for "M" epochs, and then evaluate their performance.

Do you want to unfreeze the feat-extractor at any time?

• No need to unfreeze the feature extractor.

[ekiefl]

Any guidance on this?