Skip to content

Hiding of ColumnOptions #2959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
artidoro merged 9 commits into from
Mar 19, 2019
Merged

Hiding of ColumnOptions #2959

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
dabc376
internalizing of extension that contain columnoptions
artidoro Mar 13, 2019
52bad8c
internalization of columnoptions
artidoro Mar 13, 2019
e63ad50
update the extensions to make all arguments available
artidoro Mar 14, 2019
2e27634
updated samples or moved to test folder to work with changes
artidoro Mar 14, 2019
05909ad
other miscellaneous changes to make it compile
artidoro Mar 14, 2019
b24f9f5
review comments
artidoro Mar 19, 2019
7f39458
merge
artidoro Mar 19, 2019
50b99e4
review comments
artidoro Mar 19, 2019
5585ea0
review comments
artidoro Mar 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/samples/Microsoft.ML.Samples/Dynamic/Normalizer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ public static void Example()

// Composing a different pipeline if we wanted to normalize more than one column at a time.
// Using log scale as the normalization mode.
var multiColPipeline = ml.Transforms.Normalize(NormalizingEstimator.NormalizationMode.LogMeanVariance, new ColumnOptions[] { ("LogInduced", "Induced"), ("LogSpontaneous", "Spontaneous") });
var multiColPipeline = ml.Transforms.Normalize("LogInduced", "Induced", NormalizingEstimator.NormalizationMode.LogMeanVariance)
.Append(ml.Transforms.Normalize("LogSpontaneous", "Spontaneous", NormalizingEstimator.NormalizationMode.LogMeanVariance));
// The transformed data.
var multiColtransformer = multiColPipeline.Fit(trainData);
var multiColtransformedData = multiColtransformer.Transform(trainData);
Expand Down
4 changes: 2 additions & 2 deletions docs/samples/Microsoft.ML.Samples/Dynamic/TensorFlow/TextClassification.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,10 @@ public static void Example()
};

var model = mlContext.Transforms.Text.TokenizeIntoWords("TokenizedWords", "Sentiment_Text")
.Append(mlContext.Transforms.Conversion.MapValue(lookupMap, "Words", "Ids", new ColumnOptions[] { ("VariableLenghtFeatures", "TokenizedWords") }))
.Append(mlContext.Transforms.Conversion.MapValue("VariableLenghtFeatures", lookupMap, "Words", "Ids", "TokenizedWords"))
.Append(mlContext.Transforms.CustomMapping(ResizeFeaturesAction, "Resize"))
.Append(tensorFlowModel.ScoreTensorFlowModel(new[] { "Prediction/Softmax" }, new[] { "Features" }))
.Append(mlContext.Transforms.CopyColumns(("Prediction", "Prediction/Softmax")))
.Append(mlContext.Transforms.CopyColumns("Prediction", "Prediction/Softmax"))
.Fit(dataView);
var engine = mlContext.Model.CreatePredictionEngine<IMDBSentiment, OutputScores>(model);

Expand Down
4 changes: 2 additions & 2 deletions docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/ImageAnalytics/ConvertToGrayScale.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ public static void Example()

var imagesFolder = Path.GetDirectoryName(imagesDataFile);
// Image loading pipeline.
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, ("ImageObject", "ImagePath"))
.Append(mlContext.Transforms.ConvertToGrayscale(("Grayscale", "ImageObject")));
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, "ImageObject", "ImagePath")
.Append(mlContext.Transforms.ConvertToGrayscale("Grayscale", "ImageObject"));

var transformedData = pipeline.Fit(data).Transform(data);

Expand Down
2 changes: 1 addition & 1 deletion docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/ImageAnalytics/DnnFeaturizeImage.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ public static void Example()
// Installing the Microsoft.ML.DNNImageFeaturizer packages copies the models in the
// `DnnImageModels` folder.
// Image loading pipeline.
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, ("ImageObject", "ImagePath"))
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, "ImageObject", "ImagePath")
.Append(mlContext.Transforms.ResizeImages("ImageObject", imageWidth: 224, imageHeight: 224))
.Append(mlContext.Transforms.ExtractPixels("Pixels", "ImageObject"))
.Append(mlContext.Transforms.DnnFeaturizeImage("FeaturizedImage", m => m.ModelSelector.ResNet18(mlContext, m.OutputColumn, m.InputColumn), "Pixels"));
Expand Down
2 changes: 1 addition & 1 deletion docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/ImageAnalytics/ExtractPixels.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ public static void Example()

var imagesFolder = Path.GetDirectoryName(imagesDataFile);
// Image loading pipeline.
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, ("ImageObject", "ImagePath"))
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, "ImageObject", "ImagePath")
.Append(mlContext.Transforms.ResizeImages("ImageObject", imageWidth: 100, imageHeight: 100 ))
.Append(mlContext.Transforms.ExtractPixels("Pixels", "ImageObject"));

Expand Down
2 changes: 1 addition & 1 deletion docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/ImageAnalytics/LoadImages.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ public static void Example()

var imagesFolder = Path.GetDirectoryName(imagesDataFile);
// Image loading pipeline.
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, ("ImageReal", "ImagePath"));
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, "ImageReal", "ImagePath");
var transformedData = pipeline.Fit(data).Transform(data);

// The transformedData IDataView contains the loaded images now
Expand Down
2 changes: 1 addition & 1 deletion docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/ImageAnalytics/ResizeImages.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ public static void Example()

var imagesFolder = Path.GetDirectoryName(imagesDataFile);
// Image loading pipeline.
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, ("ImageReal", "ImagePath"))
var pipeline = mlContext.Transforms.LoadImages(imagesFolder, "ImageReal", "ImagePath")
.Append(mlContext.Transforms.ResizeImages("ImageReal", imageWidth: 100, imageHeight: 100));


Expand Down
5 changes: 2 additions & 3 deletions ...ojection/VectorWhitenWithColumnOptions.cs → ...rms/Projection/VectorWhitenWithOptions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

namespace Microsoft.ML.Samples.Dynamic
{
public sealed class VectorWhitenWithColumnOptions
public sealed class VectorWhitenWithOptions
{
/// This example requires installation of additional nuget package <a href="https://www.nuget.org/packages/Microsoft.ML.Mkl.Components/">Microsoft.ML.Mkl.Components</a>.
public static void Example()
Expand Down Expand Up @@ -39,8 +39,7 @@ public static void Example()


// A pipeline to project Features column into white noise vector.
var whiteningPipeline = ml.Transforms.VectorWhiten(new Transforms.VectorWhiteningEstimator.ColumnOptions(
nameof(SamplesUtils.DatasetUtils.SampleVectorOfNumbersData.Features), kind: Transforms.WhiteningKind.PrincipalComponentAnalysis, rank: 4));
var whiteningPipeline = ml.Transforms.VectorWhiten(nameof(SamplesUtils.DatasetUtils.SampleVectorOfNumbersData.Features), kind: Transforms.WhiteningKind.PrincipalComponentAnalysis, rank: 4);
// The transformed (projected) data.
var transformedData = whiteningPipeline.Fit(trainData).Transform(trainData);
// Getting the data of the newly created column, so we can preview it.
Expand Down
6 changes: 3 additions & 3 deletions docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/ReplaceMissingValues.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
using System.Collections.Generic;
using System.Linq;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.MissingValueReplacingEstimator.ColumnOptions;
using Microsoft.ML.Transforms;

namespace Microsoft.ML.Samples.Dynamic
{
Expand All @@ -25,7 +25,7 @@ public static void Example()
var data = mlContext.Data.LoadFromEnumerable(samples);

// ReplaceMissingValues is used to create a column where missing values are replaced according to the ReplacementMode.
var meanPipeline = mlContext.Transforms.ReplaceMissingValues("MissingReplaced", "Features", ReplacementMode.Mean);
var meanPipeline = mlContext.Transforms.ReplaceMissingValues("MissingReplaced", "Features", MissingValueReplacingEstimator.ReplacementMode.Mean);

// Now we can transform the data and look at the output to confirm the behavior of the estimator.
// This operation doesn't actually evaluate data until we read the data below.
Expand All @@ -36,7 +36,7 @@ public static void Example()
var meanRowEnumerable = mlContext.Data.CreateEnumerable<SampleDataTransformed>(meanTransformedData, reuseRowObject: false);

// ReplaceMissingValues is used to create a column where missing values are replaced according to the ReplacementMode.
var defaultPipeline = mlContext.Transforms.ReplaceMissingValues("MissingReplaced", "Features", ReplacementMode.DefaultValue);
var defaultPipeline = mlContext.Transforms.ReplaceMissingValues("MissingReplaced", "Features", MissingValueReplacingEstimator.ReplacementMode.DefaultValue);

// Now we can transform the data and look at the output to confirm the behavior of the estimator.
// This operation doesn't actually evaluate data until we read the data below.
Expand Down
22 changes: 7 additions & 15 deletions docs/samples/Microsoft.ML.Samples/Dynamic/ValueMapping.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,24 +37,16 @@ public static void Example()
// 35.0 1.0 6-11yrs 1.0 3.0 32.0 5.0 ...

// If the list of keys and values are known, they can be passed to the API. The ValueMappingEstimator can also get the mapping through an IDataView
// Creating a list of keys based on the Education values from the dataset.
var educationKeys = new List<string>()
// Creating a list of key-value pairs based on the Education values from the dataset.
var educationKeyValuePairs = new List<KeyValuePair<string, string>>()
{
"0-5yrs",
"6-11yrs",
"12+yrs"
new KeyValuePair<string, string>("0-5yrs", "Undergraduate"),
new KeyValuePair<string, string>("6-11yrs", "Postgraduate"),
new KeyValuePair<string, string>("12+yrs", "Postgraduate")
};

// Creating a list of associated values that will map respectively to each educationKey
var educationValues = new List<string>()
{
"Undergraduate",
"Postgraduate",
"Postgraduate"
};


// Constructs the ValueMappingEstimator making the ML.net pipeline
var pipeline = mlContext.Transforms.Conversion.MapValue(educationKeys, educationValues, ("EducationCategory", "Education"));
var pipeline = mlContext.Transforms.Conversion.MapValue("EducationCategory", educationKeyValuePairs, "Education");

// Fits the ValueMappingEstimator and transforms the data converting the Education to EducationCategory.
IDataView transformedData = pipeline.Fit(trainData).Transform(trainData);
Expand Down
22 changes: 7 additions & 15 deletions docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingFloatToString.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,24 +28,16 @@ public static void Example()
IDataView trainData = mlContext.Data.LoadFromEnumerable(data);

// If the list of keys and values are known, they can be passed to the API. The ValueMappingEstimator can also get the mapping through an IDataView
// Creating a list of keys based on the induced value from the dataset
var temperatureKeys = new List<float>()
// Creating a list of key-value pairs based on the induced value from the dataset
var temperatureKeyValuePairs = new List<KeyValuePair<float, string>>()
{
36.0f,
35.0f,
34.0f
new KeyValuePair<float, string>(36.0f, "T1"),
new KeyValuePair<float, string>(35.0f, "T2"),
new KeyValuePair<float, string>(34.0f, "T3")
};

// Creating a list of values, these strings will map accordingly to each key.
var classificationValues = new List<string>()
{
"T1",
"T2",
"T3"
};


// Constructs the ValueMappingEstimator making the ML.net pipeline
var pipeline = mlContext.Transforms.Conversion.MapValue(temperatureKeys, classificationValues, ("TemperatureCategory", "Temperature"));
var pipeline = mlContext.Transforms.Conversion.MapValue("TemperatureCategory", temperatureKeyValuePairs, "Temperature");

// Fits the ValueMappingEstimator and transforms the data adding the TemperatureCategory column.
IDataView transformedData = pipeline.Fit(trainData).Transform(trainData);
Expand Down
20 changes: 6 additions & 14 deletions docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToArray.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,24 +31,16 @@ public static void Example()
IDataView trainData = mlContext.Data.LoadFromEnumerable(data);

// If the list of keys and values are known, they can be passed to the API. The ValueMappingEstimator can also get the mapping through an IDataView
// Creating a list of keys based on the Education values from the dataset
var educationKeys = new List<string>()
// Creating a list of key-value pairs based on the Education values from the dataset
var educationKeyValuePairs = new List<KeyValuePair<string, int[]>>()
{
"0-5yrs",
"6-11yrs",
"12+yrs"
};

// Sample list of associated array values
var educationValues = new List<int[]>()
{
new int[] { 1,2,3 },
new int[] { 5,6,7 },
new int[] { 42,32,64 }
new KeyValuePair<string, int[]>("0-5yrs", new int[] { 1,2,3 }),
new KeyValuePair<string, int[]>("6-11yrs", new int[] { 1,2,3 }),
new KeyValuePair<string, int[]>("12+yrs", new int[] { 1,2,3 })
};

// Constructs the ValueMappingEstimator making the ML.net pipeline
var pipeline = mlContext.Transforms.Conversion.MapValue<string, int>(educationKeys, educationValues, ("EducationFeature", "Education"));
var pipeline = mlContext.Transforms.Conversion.MapValue<string, int>("EducationFeature", educationKeyValuePairs, "Education");

// Fits the ValueMappingEstimator and transforms the data adding the EducationFeature column.
IDataView transformedData = pipeline.Fit(trainData).Transform(trainData);
Expand Down
22 changes: 7 additions & 15 deletions docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,28 +34,20 @@ public static void Example()
IEnumerable<SamplesUtils.DatasetUtils.SampleInfertData> data = SamplesUtils.DatasetUtils.GetInfertData();
IDataView trainData = mlContext.Data.LoadFromEnumerable(data);

// Creating a list of keys based on the Education values from the dataset
// Creating a list of key-value pairs based on the Education values from the dataset
// These lists are created by hand for the demonstration, but the ValueMappingEstimator does take an IEnumerable.
var educationKeys = new List<string>()
var educationKeyValuePairs = new List<KeyValuePair<string,string>>()
{
"0-5yrs",
"6-11yrs",
"12+yrs"
};

// Creating a list of values that are sample strings. These will be converted to KeyTypes
var educationValues = new List<string>()
{
"Undergraduate",
"Postgraduate",
"Postgraduate"
new KeyValuePair<string,string>("0-5yrs", "Undergraduate"),
new KeyValuePair<string,string>("6-11yrs", "Postgraduate"),
new KeyValuePair<string,string>("12+yrs", "Postgraduate")
};

@TomFinley TomFinley Mar 19, 2019 •
edited by artidoro
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you have this code here.

var educationKeyValuePairs = new List<KeyValuePair<string,string>>()
{
    new KeyValuePair<string,string>("0-5yrs", "Undergraduate"),
    new KeyValuePair<string,string>("6-11yrs", "Postgraduate"),
    new KeyValuePair<string,string>("12+yrs", "Postgraduate")
};

I mean, that works I guess, it is technically not incorrect, but the most obvious thing that comes to mind when I hear something that implements an IEnumerable<KeyValuePair<A, B>> is Dictionary<A, B>. So this could have been written like so.

var educationMap = new Dictionary<string, string>();
educationMap["0-5yrs"] = "Undergraduate";
educationMap["6-11yrs"] = "Postgraduate";
educationMap["12+yrs"] = "Postgraduate";

Then you just use educationMap as you use your educationKeyValuePairs.

Not saying you have to do it that way, but given that this is a sample and meant to be read, it might be nice if we kept "code oddity" to a minimum. (Though I don't insist on this, we can fix samples later. Only if you have time.) #Resolved


// Generate the ValueMappingEstimator that will output KeyTypes even though our values are strings.
// The KeyToValueMappingEstimator is added to provide a reverse lookup of the KeyType, converting the KeyType value back
// to the original value.
var pipeline = mlContext.Transforms.Conversion.MapValue<string, string>(educationKeys, educationValues, true, ("EducationKeyType", "Education"))
.Append(mlContext.Transforms.Conversion.MapKeyToValue(("EducationCategory", "EducationKeyType")));
var pipeline = mlContext.Transforms.Conversion.MapValue("EducationKeyType", educationKeyValuePairs, "Education", true)
.Append(mlContext.Transforms.Conversion.MapKeyToValue("EducationCategory", "EducationKeyType"));

// Fits the ValueMappingEstimator and transforms the data adding the EducationKeyType column.
IDataView transformedData = pipeline.Fit(trainData).Transform(trainData);
Expand Down
Loading