Skip to content

Cancellation in Image Classification (fixes #4632) #4650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
antoniovs1029 merged 11 commits into from
Jan 17, 2020
Merged

Cancellation in Image Classification (fixes #4632) #4650

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
fc139f7
Modified Program.cs
antoniovs1029 Jan 11, 2020
7147772
Added other sample for Image Classification
antoniovs1029 Jan 11, 2020
3a7a0bb
Added sample to test cancelling method
antoniovs1029 Jan 13, 2020
9a5afbd
Added CheckAlive() Checkpoints
antoniovs1029 Jan 13, 2020
052a6e4
Restore Samples' Program.cs back to original and delete test sample
antoniovs1029 Jan 13, 2020
8785b8f
Samples back to original state
antoniovs1029 Jan 13, 2020
f6042ed
Reset to original state of samples
antoniovs1029 Jan 13, 2020
eef112c
Removed checkpoints from unnecessary places
antoniovs1029 Jan 14, 2020
54d0309
Adding CheckAlive method with exception handling
antoniovs1029 Jan 14, 2020
53b737b
Added checkpoints with new CheckAlive method
antoniovs1029 Jan 14, 2020
681c13c
Removed unused exception variable "e"
antoniovs1029 Jan 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions src/Microsoft.ML.Vision/ImageClassificationTrainer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -835,6 +835,7 @@ private void CacheFeaturizedImagesToDisk(IDataView input, string labelColumnName
metrics.Bottleneck.DatasetUsed = dataset;
while (cursor.MoveNext())
{
CheckAlive();
labelGetter(ref label);
imageGetter(ref image);
if (image.Length <= 0)
Expand Down Expand Up @@ -888,6 +889,7 @@ private void CreateFeaturizedCacheFile(string cacheFilePath, int examples, int f

foreach (var row in featurizedImages)
{
CheckAlive();
writer.WriteLine(row.Item1 + "," + string.Join(",", row.Item2));
labels[0] = row.Item1;
for (int index = 0; index < sizeof(long); index++)
Expand Down Expand Up @@ -992,6 +994,7 @@ private void TrainAndEvaluateClassificationLayer(string trainBottleneckFilePath,

for (int epoch = 0; epoch < epochs; epoch += 1)
{
CheckAlive();
// Train.
TrainAndEvaluateClassificationLayerCore(epoch, learningRate, featureFileStartOffset,
metrics, labelTensorShape, featureTensorShape, batchSize,
Expand Down Expand Up @@ -1119,6 +1122,19 @@ private void TrainAndEvaluateClassificationLayerCore(int epoch, float learningRa
}
}

private void CheckAlive()
{
try
{
Host.CheckAlive();
}
catch(OperationCanceledException)
{
TryCleanupTemporaryWorkspace();
throw;
}
}

private void TryCleanupTemporaryWorkspace()
{
if (_cleanupWorkspace && Directory.Exists(_options.WorkspacePath))
Expand Down