Skip to content

Fixed & Reactivated AutoFitImageClassificationTrainTest hanging by freeing old Tensor objects #4978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mstfbl wants to merge 25 commits into from
Closed

Fixed & Reactivated AutoFitImageClassificationTrainTest hanging by freeing old Tensor objects #4978

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
f4a910e
Free Tensor objects in finally statement
mstfbl Mar 27, 2020
4f8475c
Update RunnerUtil.cs
mstfbl Mar 27, 2020
f7e1337
Re-enable AutoFitImageClassificationTrainTest after fix
mstfbl Mar 30, 2020
83ad312
Added IDisposable support to ModelContainer & corrected name of model…
mstfbl Apr 7, 2020
b366fbe
Corrected name of modelContainer in used cases
mstfbl Apr 7, 2020
cb22b21
Clean up Tensor objects through finalizer/destructor of ImageClassifi…
mstfbl Apr 9, 2020
eefa76f
Dispose ExperimentResult objects at the end
mstfbl Apr 9, 2020
45681b4
Dispose only Tensor objects in models
mstfbl Apr 10, 2020
fbd3fd9
Don't free BestModel models
mstfbl Apr 12, 2020
2816ced
Merge remote-tracking branch 'upstream/master' into AutoFitImageClass…
mstfbl Apr 12, 2020
7dad242
Throw Exception if model is trying to be accessed after disposal
mstfbl Apr 14, 2020
1488d0c
Initialize IsModelDisposed inside constructors
mstfbl Apr 14, 2020
78bba9c
Model always written to disk, no longer stored in memory, simplify mo…
mstfbl Apr 16, 2020
bf84823
Model always written to disk, no longer stored in memory, simplify mo…
mstfbl Apr 16, 2020
15b6135
Merge branch 'AutoFitImageClassificationTrainTest-memoryFix' of https…
mstfbl Apr 16, 2020
087c0d5
Update ModelContainer.cs
mstfbl Apr 16, 2020
476bc0e
Added descriptions for Dispose functions and "using" instead of direc…
mstfbl Apr 16, 2020
6234fe5
Merge remote-tracking branch 'upstream/master' into AutoFitImageClass…
mstfbl May 29, 2020
ef17073
Added disposable/finalizer
mstfbl May 29, 2020
a16fccf
Run specific test 100 iterations
mstfbl May 31, 2020
3c6045f
Update AutoFitTests.cs
mstfbl Jun 1, 2020
289bc12
Update .vsts-dotnet-ci.yml
mstfbl Jun 4, 2020
5fb9450
Update AutoFitTests.cs
mstfbl Jun 4, 2020
7668f04
Update AutoFitTests.cs
mstfbl Jun 4, 2020
cb90b8f
Update AutoFitTests.cs
mstfbl Jun 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/Microsoft.ML.AutoML/Experiment/Runners/CrossValRunner.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ public CrossValRunner(MLContext context,
var modelFileInfo = RunnerUtil.GetModelFileInfo(modelDirectory, iterationNum, i + 1);
var trainResult = RunnerUtil.TrainAndScorePipeline(_context, pipeline, _trainDatasets[i], _validDatasets[i],
_labelColumn, _metricsAgent, _preprocessorTransforms?[i], modelFileInfo, _modelInputSchema, _logger);
trainResults.Add(new SuggestedPipelineTrainResult<TMetrics>(trainResult.model, trainResult.metrics, trainResult.exception, trainResult.score));
trainResults.Add(new SuggestedPipelineTrainResult<TMetrics>(trainResult.modelContainer, trainResult.metrics, trainResult.exception, trainResult.score));
}

var avgScore = CalcAverageScore(trainResults.Select(r => r.Score));
Expand Down
2 changes: 1 addition & 1 deletion src/Microsoft.ML.AutoML/Experiment/Runners/RunnerUtil.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ namespace Microsoft.ML.AutoML
{
internal static class RunnerUtil
{
public static (ModelContainer model, TMetrics metrics, Exception exception, double score)
public static (ModelContainer modelContainer, TMetrics metrics, Exception exception, double score)
TrainAndScorePipeline<TMetrics>(MLContext context,
SuggestedPipeline pipeline,
IDataView trainData,
Expand Down
2 changes: 1 addition & 1 deletion src/Microsoft.ML.AutoML/Experiment/Runners/TrainValidateRunner.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public TrainValidateRunner(MLContext context,
trainResult.score,
trainResult.exception == null,
trainResult.metrics,
trainResult.model,
trainResult.modelContainer,
trainResult.exception);
var runDetail = suggestedPipelineRunDetail.ToIterationResult(_preFeaturizer);
return (suggestedPipelineRunDetail, runDetail);
Expand Down
4 changes: 4 additions & 0 deletions src/Microsoft.ML.Vision/ImageClassificationTrainer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1509,5 +1509,9 @@ public void Dispose()

_isDisposed = true;
}

// Finalizer to clean-up remaining Tensor objects generated in TensorFlow C libraries
// not cleaned up by GC
~ImageClassificationModelParameters() => Dispose();
Comment thread
mstfbl marked this conversation as resolved.
Outdated
}
}
2 changes: 0 additions & 2 deletions test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,6 @@ public void AutoFitMultiTest()
}

[TensorFlowFact]
//Skipping test temporarily. This test will be re-enabled once the cause of failures has been determined
[Trait("Category", "SkipInCI")]
public void AutoFitImageClassificationTrainTest()
{
var context = new MLContext(seed: 1);
Expand Down