Skip to content

Added categorical value support for PredictedLabel for the Image Classification Transfer Learning example. Addresses Issue #4169 #4228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
codemzs merged 31 commits into from
Oct 3, 2019
Merged

Added categorical value support for PredictedLabel for the Image Classification Transfer Learning example. Addresses Issue #4169 #4228

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
8d07401
Attempt on Issue 4169
mstfbl Sep 18, 2019
d80e23d
Further work on Issue_4169
mstfbl Sep 20, 2019
7b239b0
Temporary change for inquiry
mstfbl Sep 24, 2019
33e71ae
Pushing changes for inquiry
mstfbl Sep 26, 2019
2b1ab69
Implemented PredictedLabel as Categorical value (String/int). Now wor…
mstfbl Sep 26, 2019
7254a7e
Merge branch 'Issue_4169' of https://github.com/mstfbl/machinelearnin…
mstfbl Sep 26, 2019
1a43cc7
Added tests for Prediction Engine
mstfbl Sep 26, 2019
d39f98b
Removed the forwarding of DataViewSchema to the TrySinglePrediction f…
mstfbl Sep 26, 2019
8cf2b7c
Minor performance upgrade to avoid the array bounds checkl
mstfbl Sep 27, 2019
c4182fb
Update ImageClassificationTransform.cs
mstfbl Sep 30, 2019
d490383
Updated tests
mstfbl Oct 2, 2019
01c77c9
Merge branch 'Issue_4169' of https://github.com/mstfbl/machinelearnin…
mstfbl Oct 2, 2019
380d8bf
Revert "Merge branch 'Issue_4169' of https://github.com/mstfbl/machin…
mstfbl Oct 2, 2019
2af03ca
Revert "Updated tests"
mstfbl Oct 2, 2019
2c05ba8
Updated test files and corrected variable spellings
mstfbl Oct 2, 2019
6356665
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
mstfbl Oct 2, 2019
ed928c6
Update ImageClassificationTransform.cs
mstfbl Oct 2, 2019
f7f8253
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
mstfbl Oct 2, 2019
bd42f0a
Revert "Merge branch 'master' of https://github.com/dotnet/machinelea…
mstfbl Oct 2, 2019
f851791
Merge branch 'master' into Issue_4169
mstfbl Oct 2, 2019
643fb58
Deleted unused _outputTypes
mstfbl Oct 2, 2019
b6cfeda
Update ImageClassificationTransform.cs
mstfbl Oct 2, 2019
cca4fb8
Added test case to check the matching of predicted labels
mstfbl Oct 2, 2019
1c4c5dc
Update ImageClassificationTransform.cs
mstfbl Oct 2, 2019
2301a96
Update TensorflowTests.cs
mstfbl Oct 2, 2019
26e2ae1
Update TensorflowTests.cs
mstfbl Oct 2, 2019
94c0423
Update TensorflowTests.cs
mstfbl Oct 3, 2019
109879b
Fixed test case and off-by-one predictedLabel error
mstfbl Oct 3, 2019
6ff4c64
Merge remote-tracking branch 'upstream/master' into Issue_4169
mstfbl Oct 3, 2019
15c4654
Removed comments
mstfbl Oct 3, 2019
0573ef3
Replacing Path.Join with Path.Combine due to build error with Path.Join
mstfbl Oct 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 7 additions & 13 deletions ...osoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,8 @@ public static void Example()
batchSize: 10,
learningRate: 0.01f,
metricsCallback: (metrics) => Console.WriteLine(metrics),
validationSet: testDataset);

validationSet: testDataset)
.Append(mlContext.Transforms.Conversion.MapKeyToValue(outputColumnName: "PredictedLabel", inputColumnName: "Label"));
Comment thread
mstfbl marked this conversation as resolved.
Outdated

Console.WriteLine("*** Training the image classification model with " +
"DNN Transfer Learning on top of the selected pre-trained " +
Expand All @@ -96,12 +96,8 @@ public static void Example()

EvaluateModel(mlContext, testDataset, loadedModel);

VBuffer<ReadOnlyMemory<char>> keys = default;
loadedModel.GetOutputSchema(schema)["Label"].GetKeyValues(ref keys);

watch = System.Diagnostics.Stopwatch.StartNew();
TrySinglePrediction(fullImagesetFolderPath, mlContext, loadedModel,
keys.DenseValues().ToArray());
TrySinglePrediction(fullImagesetFolderPath, mlContext, loadedModel);

watch.Stop();
elapsedMs = watch.ElapsedMilliseconds;
Expand All @@ -119,12 +115,11 @@ public static void Example()
}

private static void TrySinglePrediction(string imagesForPredictions,
MLContext mlContext, ITransformer trainedModel,
ReadOnlyMemory<char>[] originalLabels)
MLContext mlContext, ITransformer trainedModel)
{
// Create prediction function to try one prediction
var predictionEngine = mlContext.Model
.CreatePredictionEngine<ImageData, ImagePrediction>(trainedModel);
.CreatePredictionEngine<ImageData, ImagePrediction>(trainedModel, true);
Comment thread
mstfbl marked this conversation as resolved.
Outdated

IEnumerable<ImageData> testImages = LoadImagesFromDirectory(
imagesForPredictions, false);
Expand All @@ -135,12 +130,11 @@ private static void TrySinglePrediction(string imagesForPredictions,
};

var prediction = predictionEngine.Predict(imageToPredict);
var index = prediction.PredictedLabel;

Console.WriteLine($"ImageFile : " +
$"[{Path.GetFileName(imageToPredict.ImagePath)}], " +
$"Scores : [{string.Join(",", prediction.Score)}], " +
$"Predicted Label : {originalLabels[index]}");
$"Predicted Label : {prediction.PredictedLabel}");
}


Expand Down Expand Up @@ -300,7 +294,7 @@ public class ImagePrediction
public float[] Score;

[ColumnName("PredictedLabel")]
public UInt32 PredictedLabel;
public string PredictedLabel;
}
}
}
Expand Down
51 changes: 38 additions & 13 deletions src/Microsoft.ML.Dnn/ImageClassificationTransform.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ public sealed class ImageClassificationTransformer : RowToRowTransformerBase
private Graph Graph => _session.graph;
private readonly string[] _inputs;
private readonly string[] _outputs;
private readonly string[] _keyValueAnnotations;
private readonly string _labelColumnName;
private readonly string _finalModelPrefix;
private readonly Architecture _arch;
Expand Down Expand Up @@ -109,7 +110,7 @@ private static ImageClassificationTransformer Create(IHostEnvironment env, Model

GetModelInfo(env, ctx, out string[] inputs, out string[] outputs, out bool addBatchDimensionInput,
out string labelColumn, out string checkpointName, out Architecture arch, out string scoreColumnName,
out string predictedColumnName, out float learningRate, out int classCount, out string predictionTensorName, out string softMaxTensorName,
out string predictedColumnName, out float learningRate, out int classCount, out string[] keyValueAnnotations, out string predictionTensorName, out string softMaxTensorName,
out string jpegDataTensorName, out string resizeTensorName);

byte[] modelBytes = null;
Expand All @@ -118,7 +119,7 @@ private static ImageClassificationTransformer Create(IHostEnvironment env, Model

return new ImageClassificationTransformer(env, DnnUtils.LoadTFSession(env, modelBytes), outputs, inputs,
null, addBatchDimensionInput, 1, labelColumn, checkpointName, arch,
scoreColumnName, predictedColumnName, learningRate, null, classCount, true, predictionTensorName,
scoreColumnName, predictedColumnName, learningRate, null, classCount, keyValueAnnotations, true, predictionTensorName,
softMaxTensorName, jpegDataTensorName, resizeTensorName);

}
Expand Down Expand Up @@ -617,7 +618,7 @@ private static IRowMapper Create(IHostEnvironment env, ModelLoadContext ctx, Dat
private static void GetModelInfo(IHostEnvironment env, ModelLoadContext ctx, out string[] inputs,
out string[] outputs, out bool addBatchDimensionInput,
out string labelColumn, out string checkpointName, out Architecture arch,
out string scoreColumnName, out string predictedColumnName, out float learningRate, out int classCount, out string predictionTensorName, out string softMaxTensorName,
out string scoreColumnName, out string predictedColumnName, out float learningRate, out int classCount, out string[] keyValueAnnotations, out string predictionTensorName, out string softMaxTensorName,
out string jpegDataTensorName, out string resizeTensorName)
{
addBatchDimensionInput = ctx.Reader.ReadBoolByte();
Expand All @@ -641,6 +642,12 @@ private static void GetModelInfo(IHostEnvironment env, ModelLoadContext ctx, out
predictedColumnName = ctx.Reader.ReadString();
learningRate = ctx.Reader.ReadFloat();
classCount = ctx.Reader.ReadInt32();

env.CheckDecode(classCount > 0);
keyValueAnnotations = new string[classCount];
for (int j = 0; j < keyValueAnnotations.Length; j++)
keyValueAnnotations[j] = ctx.LoadNonEmptyString();

predictionTensorName = ctx.Reader.ReadString();
softMaxTensorName = ctx.Reader.ReadString();
jpegDataTensorName = ctx.Reader.ReadString();
Expand All @@ -650,7 +657,7 @@ private static void GetModelInfo(IHostEnvironment env, ModelLoadContext ctx, out
internal ImageClassificationTransformer(IHostEnvironment env, Session session, string[] outputColumnNames,
string[] inputColumnNames, string modelLocation,
bool? addBatchDimensionInput, int batchSize, string labelColumnName, string finalModelPrefix, Architecture arch,
string scoreColumnName, string predictedLabelColumnName, float learningRate, DataViewSchema inputSchema, int? classCount = null, bool loadModel = false,
string scoreColumnName, string predictedLabelColumnName, float learningRate, DataViewSchema inputSchema, int? classCount = null, string[] keyValueAnnotations = null, bool loadModel = false,

@yaeldekel yaeldekel Sep 24, 2019 •
edited by mstfbl
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keyValueAnnotations [](start = 150, length = 19)

The transformer can be created in one of two ways: Either by calling Fit on the estimator, or by deserializing from file. In the second case, the key values come from reading them from the model as you have written. In the first case, you need to get them from inputSchema by getting the key value annotations of the training label column. #Resolved

@yaeldekel yaeldekel Sep 24, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that it is possible that the label column does not contain key values annotations, which means that the PredictedLabel column will not have them either. You have to account for that at serialization/deserialization time (for example, by serialiazing the number of key value strings, and only then serializing the strings them selves).

In reply to: 327481050 [](ancestors = 327481050)

string predictionTensorName = null, string softMaxTensorName = null, string jpegDataTensorName = null, string resizeTensorName = null)
: base(Contracts.CheckRef(env, nameof(env)).Register(nameof(ImageClassificationTransformer)))

Expand All @@ -664,6 +671,7 @@ internal ImageClassificationTransformer(IHostEnvironment env, Session session, s
_addBatchDimensionInput = addBatchDimensionInput ?? arch == Architecture.ResnetV2101;
_inputs = inputColumnNames;
_outputs = outputColumnNames;
_keyValueAnnotations = keyValueAnnotations;
_labelColumnName = labelColumnName;
_finalModelPrefix = finalModelPrefix;
_arch = arch;
Expand All @@ -684,6 +692,9 @@ internal ImageClassificationTransformer(IHostEnvironment env, Session session, s
throw Host.ExceptSchemaMismatch(nameof(inputSchema), "label", (string)labelColumn.Name, "Key", (string)labelType.ToString());

_classCount = labelCount == 1 ? 2 : (int)labelCount;

@yaeldekel yaeldekel Sep 24, 2019 •
edited by mstfbl
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

labelCount == 1 ? 2 : (int)labelCount [](start = 30, length = 37)

Why?
/cc @codemzs
#Resolved

@codemzs codemzs Sep 29, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To account for binary classification case where in the train set all you have seen is just one class :-).

In reply to: 327483161 [](ancestors = 327483161)


//Temporary string array for key values, will replace soon
_keyValueAnnotations = new string[] { "sunflowers", "roses", "dandelion", "daisy", "tulips"};

@yaeldekel yaeldekel Sep 24, 2019 •
edited by mstfbl
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_keyValueAnnotations [](start = 16, length = 20)

It seems that the loadModel argument specifies whether we are training the model or loading it from a file, inside the if in line 718 is where you should extract the key values from the input schema. #Resolved

}
else
_classCount = classCount.Value;
Expand Down Expand Up @@ -749,6 +760,10 @@ private protected override void SaveModel(ModelSaveContext ctx)
ctx.Writer.Write(_predictedLabelColumnName);
ctx.Writer.Write(_learningRate);
ctx.Writer.Write(_classCount);

foreach (var keyValueAnnotation in _keyValueAnnotations)
ctx.SaveNonEmptyString(keyValueAnnotation);

ctx.Writer.Write(_predictionTensorName);
ctx.Writer.Write(_softmaxTensorName);
ctx.Writer.Write(_jpegDataTensorName);
Expand Down Expand Up @@ -877,8 +892,8 @@ private protected override Func<int, bool> GetDependenciesCore(Func<int, bool> a
protected override DataViewSchema.DetachedColumn[] GetOutputColumnsCore()
{
var info = new DataViewSchema.DetachedColumn[_parent._outputs.Length];
info[0] = new DataViewSchema.DetachedColumn(_parent._outputs[0], new VectorDataViewType(NumberDataViewType.Single, _parent._classCount), null);
info[1] = new DataViewSchema.DetachedColumn(_parent._outputs[1], NumberDataViewType.UInt32, null);
info[0] = new DataViewSchema.DetachedColumn(_parent._scoreColumnName, new VectorDataViewType(NumberDataViewType.Single, _parent._classCount), null);
info[1] = new DataViewSchema.DetachedColumn(_parent._predictedLabelColumnName, new KeyDataViewType(typeof(uint), _parent._classCount), ((DataViewSchema.Column)InputSchema.GetColumnOrNull(_parent._labelColumnName)).Annotations);
return info;
}
}
Expand Down Expand Up @@ -1038,6 +1053,12 @@ internal sealed class Options : TransformInputBase
[Argument(ArgumentType.Multiple | ArgumentType.Required, HelpText = "The name of the outputs", ShortName = "outputs", SortOrder = 2)]
public string[] OutputColumns;

/// <summary>
/// The names of the requested key value annotations.
/// </summary>
[Argument(ArgumentType.Multiple | ArgumentType.Required, HelpText = "The name of the key value annotations", ShortName = "annotations", SortOrder = 3)]
public string[] KeyValueAnnotations;
Comment thread
mstfbl marked this conversation as resolved.
Outdated

/// <summary>
/// The name of the label column in <see cref="IDataView"/> that will be mapped to label node in TensorFlow model.
/// </summary>
Expand Down Expand Up @@ -1166,7 +1187,7 @@ internal ImageClassificationEstimator(IHostEnvironment env, Options options, Dnn
_options = options;
_dnnModel = dnnModel;
_tfInputTypes = new[] { TF_DataType.TF_STRING };
_outputTypes = new[] { new VectorDataViewType(NumberDataViewType.Single), NumberDataViewType.UInt32.GetItemType() };
_outputTypes = new DataViewType[] { new VectorDataViewType(NumberDataViewType.Single), new KeyDataViewType(typeof(uint), 5) };
Comment thread
mstfbl marked this conversation as resolved.
Outdated

@yaeldekel yaeldekel Sep 19, 2019 •
edited by mstfbl
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_outputTypes [](start = 12, length = 12)

_outputTypes doesn't need to be an array of DataViewType. All the estimator needs to know is whether the length of the score vector is variable or not (seems to me that it is typically not variable, or at least this information can be inferred from the DnnModel).
If you get rid of this field, you will not have to guess the size of the key (which is also not needed by the estimator). #Resolved

@mstfbl mstfbl Sep 19, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good catch, thank you! Quick question, the Image Classification example ResnetV2101TransferLearningTrainTestSplit.cs runs perfectly well when I don't further define _outputTypes, i,e, delete line 69. Does this also mean than the estimator isn't using this field? #Resolved

@mstfbl mstfbl Sep 19, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I've found that the DataViewType field is needed for the pipeline in ResnetV2101TransferLearningTrainTestSplit.cs to fit properly. #Resolved

@yaeldekel yaeldekel Sep 19, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate? The only place _outputTypes is used is in GetOutputSchema (and it is actually being used incorrectly there - see my comments there). What is not working properly in the example you mentioned?

In reply to: 326329163 [](ancestors = 326329163)

@mstfbl mstfbl Sep 20, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was giving a different error when I removed _outputTypes, but as I now see that was not the bottleneck problem I was having while implementing this KeyType solution. #Resolved

@yaeldekel yaeldekel Oct 2, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The field _outputTypes can be deleted. The only place where it is used is in line 1281 where you have _outputTypes[0].GetItemType(). Instead of that you can write NumberDataViewType.Single, and then the field is not needed.

In reply to: 326229209 [](ancestors = 326229209)

@mstfbl mstfbl Oct 2, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed this, thank you for the catch! #Resolved

}

private static Options CreateArguments(DnnModel tensorFlowModel, string[] outputColumnNames, string[] inputColumnName, bool addBatchDimensionInput)
Expand Down Expand Up @@ -1196,12 +1217,16 @@ public SchemaShape GetOutputSchema(SchemaShape inputSchema)
if (col.ItemType != expectedType)
throw _host.ExceptSchemaMismatch(nameof(inputSchema), "input", input, expectedType.ToString(), col.ItemType.ToString());
}
for (var i = 0; i < _options.OutputColumns.Length; i++)
{
resultDic[_options.OutputColumns[i]] = new SchemaShape.Column(_options.OutputColumns[i],
_outputTypes[i].IsKnownSizeVector() ? SchemaShape.Column.VectorKind.Vector
: SchemaShape.Column.VectorKind.VariableVector, _outputTypes[i].GetItemType(), false);
}

resultDic[_options.OutputColumns[0]] = new SchemaShape.Column(_options.OutputColumns[0],
SchemaShape.Column.VectorKind.Vector, _outputTypes[0].GetItemType(), false);

var metadata = new List<SchemaShape.Column>();
metadata.Add(new SchemaShape.Column(AnnotationUtils.Kinds.KeyValues, SchemaShape.Column.VectorKind.Vector, TextDataViewType.Instance, false, inputSchema[1].Annotations));
Comment thread
mstfbl marked this conversation as resolved.
Outdated

resultDic[_options.OutputColumns[1]] = new SchemaShape.Column(_options.OutputColumns[1],
SchemaShape.Column.VectorKind.Scalar, NumberDataViewType.UInt32, true, new SchemaShape(metadata.ToArray()));

return new SchemaShape(resultDic.Values);
}

Expand Down