Skip to content

Cancellation in Image Classification (fixes #4632) #4650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
antoniovs1029 merged 11 commits into from
Jan 17, 2020
Merged

Cancellation in Image Classification (fixes #4632) #4650

Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
fc139f7
Modified Program.cs
antoniovs1029 Jan 11, 2020
7147772
Added other sample for Image Classification
antoniovs1029 Jan 11, 2020
3a7a0bb
Added sample to test cancelling method
antoniovs1029 Jan 13, 2020
9a5afbd
Added CheckAlive() Checkpoints
antoniovs1029 Jan 13, 2020
052a6e4
Restore Samples' Program.cs back to original and delete test sample
antoniovs1029 Jan 13, 2020
8785b8f
Samples back to original state
antoniovs1029 Jan 13, 2020
f6042ed
Reset to original state of samples
antoniovs1029 Jan 13, 2020
eef112c
Removed checkpoints from unnecessary places
antoniovs1029 Jan 14, 2020
54d0309
Adding CheckAlive method with exception handling
antoniovs1029 Jan 14, 2020
53b737b
Added checkpoints with new CheckAlive method
antoniovs1029 Jan 14, 2020
681c13c
Removed unused exception variable "e"
antoniovs1029 Jan 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/Microsoft.ML.Vision/ImageClassificationTrainer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -647,6 +647,8 @@ private protected override MulticlassPredictionTransformer<ImageClassificationMo

private protected override ImageClassificationModelParameters TrainModelCore(TrainContext trainContext)
{
Host.CheckAlive();

// Workspace directory is cleaned after training run. However, the pipeline can be re-used by calling
// fit() again after transform(), in which case we must ensure workspace directory exists. This scenario
// is typical in the case of cross-validation.
Expand Down Expand Up @@ -906,6 +908,7 @@ private void CreateFeaturizedCacheFile(string cacheFilePath, int examples, int f
private void TrainAndEvaluateClassificationLayer(string trainBottleneckFilePath,
string validationSetBottleneckFilePath)
{
Host.CheckAlive();
Contracts.Assert(validationSetBottleneckFilePath == null ||
(File.Exists(validationSetBottleneckFilePath + "_labels.bin") &&
File.Exists(validationSetBottleneckFilePath + "_features.bin")));
Expand Down Expand Up @@ -992,6 +995,7 @@ private void TrainAndEvaluateClassificationLayer(string trainBottleneckFilePath,

for (int epoch = 0; epoch < epochs; epoch += 1)
{
Host.CheckAlive();

@codemzs codemzs Jan 14, 2020 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Host.CheckAlive(); [](start = 20, length = 18)

I would just put the check in this loop and in the CreateFeaturizedCacheFile. Please also report numbers in perf differences before and after. Please remove CheckAlive from everywhere else as its not very significant and only pollutes the code. You also need to call TryCleanupTemporaryWorkspace for a graceful termination. #Closed

@antoniovs1029 antoniovs1029 Jan 14, 2020 •
edited by codemzs
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a new method "CheckAlive" to the ImageClassification trainer, with a try...catch to call TryCleanupTemporaryWorkspace when it's needed.

Also changed the places where I added the checkpoints.

I will see how to get the perf difference now. #Closed

@antoniovs1029 antoniovs1029 Jan 14, 2020 •
edited by codemzs
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I ran the ImageClassificationBench.TrainResnetV250 benchmark, with and without the changes of this PR, and they both behaved in pretty much the same way.

Without the changes this was the summary output of the benchmark:

          Method |    Mean |   Error |   StdDev | Extra Metric |
---------------- |--------:|--------:|---------:|-------------:|
 TrainResnetV250 | 41.55 s | 5.580 s | 0.3058 s |            - |

And with the changes, the summary was:

          Method |    Mean |   Error |   StdDev | Extra Metric |
---------------- |--------:|--------:|---------:|-------------:|
 TrainResnetV250 | 40.10 s | 2.723 s | 0.1493 s |            - |

So on average the version with the changes was reported to ran faster.

In any case, the CheckAlive() method is simply doing if-statements evaluations, so I don't think it can introduce meaningful performance difference (given that image classification training is a task expected to take a considerable amount of time anyway). #Closed

@antoniovs1029 antoniovs1029 Jan 17, 2020 •
edited by codemzs
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, as suggested online by @codemzs I have reran the benchmarks, but using the CIFAR-10 dataset.

Without the changes introduced in the PR the summary is as follows:

          Method |    Mean |   Error |   StdDev | Extra Metric |
---------------- |--------:|--------:|---------:|-------------:|
 TrainResnetV250 | 79.29 m | 4.850 m | 0.2658 m |            - |

With the changes:

          Method |    Mean |   Error |  StdDev | Extra Metric |
---------------- |--------:|--------:|--------:|-------------:|
 TrainResnetV250 | 78.82 m | 21.71 m | 1.190 m |            - |

So, again, my understanding is that there's some variability in the time it takes to train this model (and that's why the benchmark with the changes ran a little bit faster), and the introduction of the CheckAlive() method doesn't really have an impact on the performance of this. #Closed

// Train.
TrainAndEvaluateClassificationLayerCore(epoch, learningRate, featureFileStartOffset,
metrics, labelTensorShape, featureTensorShape, batchSize,
Expand Down