Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/Microsoft.ML.Data/Transforms/ChooseColumnsTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,18 @@ private static VersionInfo GetVersionInfo()

private const string RegistrationName = "ChooseColumns";

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="source">Name of the selected column. If this is null '<paramref name="name"/>' will be used.</param>
public ChooseColumnsTransform(IHostEnvironment env, IDataView input, string name, string source = null)

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose column transform is one of those transforms that is pretty much completely useless unless it supports multiple columns. It's not like other transforms where you can just apply the transform again to get more columns. If something isn't in the choose list, it is simply dropped -- no second chances. We need to support multiple column specification in the constructor.

However, choose columns generally as far as I know could just do with a params string[] name or something, and be done with it. Generally it isn't used to rename columns. (That is, I see xf=choose{col=Foo,Bar,Biz}, I don't see much xf=Choose{col=Foo:Bar} or something like this.) #Closed

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually wait. What is the relationship between "choose columns transform" and "keep columns transform"? I think there is a subtle difference, but it escapes me at the moment. #Closed

@zeahmed zeahmed Jul 5, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, renaming of chosen column is the difference between ChooseColumnsTransform and KeepColumnsTransform. KeepColumnsTransform does not support renaming and drops the rest of the columns. ChooseColumnsTransform does renaming as well as has other parameters like Hidden which has following options.

public enum HiddenColumnOption : byte
{
    Drop = 1,
    Keep = 2,
    Rename = 3
}

What I suggest that for convenience this transform operate in similar way as KeepColumnsTransform.

 public ChooseColumnsTransform(IHostEnvironment env, IDataView input, param string[] columnNames)
            : this(env, new Arguments(columnNames) {  }, input)
{
}

For other usage, user can invoke constructor with Arguments object. What do you suggest? #Closed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd probably just keep the non-renaming case convenient, and nuke the constructor on arguments.


In reply to: 200471726 [](ancestors = 200471726)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean that current changes are fine? If not can you please explain bit more?


In reply to: 200727095 [](ancestors = 200727095,200471726)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are good, thank you @zeahmed.


In reply to: 200743746 [](ancestors = 200743746,200727095,200471726)

: this(env, new Arguments() { Column = new[] { new Column() { Source = source ?? name, Name = name } } }, input)
{
}

/// <summary>
/// Public constructor corresponding to SignatureDataTransform.
/// </summary>
Expand Down
29 changes: 29 additions & 0 deletions src/Microsoft.ML.Data/Transforms/ConvertTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,16 @@ public bool TryUnparse(StringBuilder sb)

public class Arguments : TransformInputBase
{
public Arguments()
{

}

public Arguments(string name, string source)

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. What's this constructor for? #Closed

{
Column = new[] { new Column() { Source = source ?? name, Name = name } };
}

[Argument(ArgumentType.Multiple | ArgumentType.Required, HelpText = "New column definition(s) (optional form: name:type:src)", ShortName = "col", SortOrder = 1)]
public Column[] Column;

Expand Down Expand Up @@ -169,6 +179,25 @@ private static VersionInfo GetVersionInfo()
// This is parallel to Infos.
private readonly ColInfoEx[] _exes;

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="source">Name of the column to be converted. If this is null '<paramref name="name"/>' will be used.</param>
/// <param name="resultType">The expected type of the converted column.</param>
/// <param name="keyRange">For a key column, this defines the range of values.</param>
public ConvertTransform(IHostEnvironment env,
IDataView input,
string name,
string source = null,
DataKind? resultType = null,

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataKind? [](start = 12, length = 9)

Hi @zeahmed thanks for doing this. Quick question, why is this nullable? What happens if it is null? #Closed

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this is interesting. Unlike some other transforms, this transform is one of those where we kind of do have a required argument. Hmmm. We need to think carefully about how we want to phrase the constructor here. We may need to refine my prior idea from #380 of having the signature be string name, string source = null. That may have been an error. #Closed

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe not. Maybe just putting them before name is fine, maybe. But that's kind of weird too, since then you have the input data view, then required parameters, then names, then parameters with defaults. But maybe. #Closed

@zeahmed zeahmed Jul 5, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding resultType being null. I see following happening in the code. The decision is being made based on resultType, KeyRange and Range parameters. At the end, if everything is null DataKind.Num is the value.

So, I would suggest setting resultType = DataKind.Num or make it required.
#Resolved

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make it required myself... I sort of view API usage as similar to programmatic usage, and if we view this "convert" as similar to a cast or something, the idea that there is a default type to cast to just seems silly. Even if maybe we want to allow it in command-line/GUI world.


In reply to: 200468398 [](ancestors = 200468398)

KeyRange keyRange = null)

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyRange keyRange = null [](start = 12, length = 24)

I don't think conversions among key ranges are common enough that they are worthy of a convenience constructor. #Closed

: this(env, new Arguments(name, source) { ResultType = resultType, KeyRange = keyRange }, input)
{
}

public ConvertTransform(IHostEnvironment env, Arguments args, IDataView input)
: base(env, RegistrationName, env.CheckRef(args, nameof(args)).Column,
input, null)
Expand Down
12 changes: 12 additions & 0 deletions src/Microsoft.ML.Data/Transforms/DropSlotsTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,18 @@ public ColInfoEx(SlotDropper slotDropper, bool suppressed, ColumnType typeDst, i

private readonly ColInfoEx[] _exes;

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="source">Name of the input column. If this is null '<paramref name="name"/>' will be used.</param>
public DropSlotsTransform(IHostEnvironment env, IDataView input, string name, string source = null)

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is a transform that works well for conveniences. Certainly this specific constructor, which really does nothing, is not something we need right now. I'd rather not do drop slots for now. #Closed

: this(env, new Arguments() { Column = new[] { new Column() { Source = source ?? name, Name = name } } }, input)
{
}

/// <summary>
/// Public constructor corresponding to SignatureDataTransform.
/// </summary>
Expand Down
22 changes: 20 additions & 2 deletions src/Microsoft.ML.Data/Transforms/GenerateNumberTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -77,16 +77,22 @@ private bool TryParse(string str)
}
}

private static class Defaults
{
public const bool UseCounter = false;
public const uint Seed = 42;
}

public sealed class Arguments : TransformInputBase
{
[Argument(ArgumentType.Multiple | ArgumentType.Required, HelpText = "New column definition(s) (optional form: name:seed)", ShortName = "col", SortOrder = 1)]
public Column[] Column;

[Argument(ArgumentType.AtMostOnce, HelpText = "Use an auto-incremented integer starting at zero instead of a random number", ShortName = "cnt")]
public bool UseCounter;
public bool UseCounter = Defaults.UseCounter;

[Argument(ArgumentType.AtMostOnce, HelpText = "The random seed")]
public uint Seed = 42;
public uint Seed = Defaults.Seed;
}

private sealed class Bindings : ColumnBindingsBase
Expand Down Expand Up @@ -250,6 +256,18 @@ private static VersionInfo GetVersionInfo()

private const string RegistrationName = "GenerateNumber";

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="useCounter">Use an auto-incremented integer starting at zero instead of a random number.</param>
public GenerateNumberTransform(IHostEnvironment env, IDataView input, string name, bool useCounter = Defaults.UseCounter)
: this(env, new Arguments() { Column = new[] { new Column() { Name = name } }, UseCounter = useCounter }, input)
{
}

/// <summary>
/// Public constructor corresponding to SignatureDataTransform.
/// </summary>
Expand Down
49 changes: 45 additions & 4 deletions src/Microsoft.ML.Data/Transforms/HashTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -33,26 +33,48 @@ public sealed class HashTransform : OneToOneTransformBase, ITransformTemplate
public const int NumBitsMin = 1;
public const int NumBitsLim = 32;

private static class Defaults
{
public const int HashBits = NumBitsLim - 1;
public const uint Seed = 314489979;
public const bool Ordered = false;
public const int InvertHash = 0;
}

public sealed class Arguments
{
public Arguments()
{

}

public Arguments(string name, string source)

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could see that in some circumstances, maybe having a specialized constructor on arguments could be useful, but why is it helpful here? #Closed

{
Column = new[] { new Column(){
Source = source ?? name,
Name = name
}
};
}

[Argument(ArgumentType.Multiple, HelpText = "New column definition(s) (optional form: name:src)", ShortName = "col",
SortOrder = 1)]
public Column[] Column;

[Argument(ArgumentType.AtMostOnce, HelpText = "Number of bits to hash into. Must be between 1 and 31, inclusive",
ShortName = "bits", SortOrder = 2)]
public int HashBits = NumBitsLim - 1;
public int HashBits = Defaults.HashBits;

[Argument(ArgumentType.AtMostOnce, HelpText = "Hashing seed")]
public uint Seed = 314489979;
public uint Seed = Defaults.Seed;

[Argument(ArgumentType.AtMostOnce, HelpText = "Whether the position of each term should be included in the hash",
ShortName = "ord")]
public bool Ordered;
public bool Ordered = Defaults.Ordered;

[Argument(ArgumentType.AtMostOnce, HelpText = "Limit the number of keys used to generate the slot name to this many. 0 means no invert hashing, -1 means no limit.",
ShortName = "ih")]
public int InvertHash;
public int InvertHash = Defaults.InvertHash;
}

public sealed class Column : OneToOneColumn
Expand Down Expand Up @@ -234,6 +256,25 @@ public override void Save(ModelSaveContext ctx)
TextModelHelper.SaveAll(Host, ctx, Infos.Length, _keyValues);
}

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="source">Name of the column to be transformed. If this is null '<paramref name="name"/>' will be used.</param>
/// <param name="hashBits">Number of bits to hash into. Must be between 1 and 31, inclusive.</param>
/// <param name="invertHash">Limit the number of keys used to generate the slot name to this many. 0 means no invert hashing, -1 means no limit.</param>
public HashTransform(IHostEnvironment env,
IDataView input,
string name,
string source = null,
int hashBits = Defaults.HashBits,
int invertHash = Defaults.InvertHash)
: this(env, new Arguments(name, source) { HashBits = hashBits, InvertHash = invertHash }, input)
{
}

public HashTransform(IHostEnvironment env, Arguments args, IDataView input)
: base(Contracts.CheckRef(env, nameof(env)), RegistrationName, env.CheckRef(args, nameof(args)).Column,
input, TestType)
Expand Down
13 changes: 13 additions & 0 deletions src/Microsoft.ML.Data/Transforms/KeyToValueTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,19 @@ private static VersionInfo GetVersionInfo()
private readonly ColumnType[] _types;
private KeyToValueMap[] _kvMaps;

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="source">Name of the input column. If this is null '<paramref name="name"/>' will be used.</param>
public KeyToValueTransform(IHostEnvironment env, IDataView input, string name, string source = null)

@Ivanidzo4ka Ivanidzo4ka Jul 6, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IHostEnvironment env, IDataView input, string name, string source = null [](start = 35, length = 72)

half of your files formatted in this way, half is one parameter for each line, why? #Closed

@zeahmed zeahmed Jul 6, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the line is going to be long enough to fit into the view, I format every parameter on separate line otherwise not.

For this particular, its fine to have it on one line. #Closed

: this(env, new Arguments() { Column = new[] { new Column() { Source = source ?? name, Name = name } } }, input)
{
}


/// <summary>
/// Public constructor corresponding to SignatureDataTransform.
/// </summary>
Expand Down
24 changes: 23 additions & 1 deletion src/Microsoft.ML.Data/Transforms/KeyToVectorTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -70,14 +70,19 @@ public bool TryUnparse(StringBuilder sb)
}
}

private static class Defaults
{
public const bool Bag = false;
}

public sealed class Arguments
{
[Argument(ArgumentType.Multiple, HelpText = "New column definition(s) (optional form: name:src)", ShortName = "col", SortOrder = 1)]
public Column[] Column;

[Argument(ArgumentType.AtMostOnce,
HelpText = "Whether to combine multiple indicator vectors into a single bag vector instead of concatenating them. This is only relevant when the input is a vector.")]
public bool Bag;
public bool Bag = Defaults.Bag;
}

internal const string Summary = "Converts a key column to an indicator vector.";
Expand Down Expand Up @@ -112,6 +117,23 @@ private static VersionInfo GetVersionInfo()
private readonly bool[] _concat;
private readonly VectorType[] _types;

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="source">Name of the input column. If this is null '<paramref name="name"/>' will be used.</param>
/// <param name="bag">Whether to combine multiple indicator vectors into a single bag vector instead of concatenating them. This is only relevant when the input is a vector.</param>
public KeyToVectorTransform(IHostEnvironment env,
IDataView input,
string name,
string source = null,
bool bag = Defaults.Bag)
: this(env, new Arguments() { Column = new[] { new Column() { Source = source ?? name, Name = name } }, Bag = bag }, input)
{
}

/// <summary>
/// Public constructor corresponding to SignatureDataTransform.
/// </summary>
Expand Down
12 changes: 12 additions & 0 deletions src/Microsoft.ML.Data/Transforms/LabelConvertTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,18 @@ private static VersionInfo GetVersionInfo()
private const string RegistrationName = "LabelConvert";
private VectorType _slotType;

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="source">Name of the input column. If this is null '<paramref name="name"/>' will be used.</param>
public LabelConvertTransform(IHostEnvironment env, IDataView input, string name, string source = null)
: this(env, new Arguments() { Column = new[] { new Column() { Source = source ?? name, Name = name } } }, input)
{
}

public LabelConvertTransform(IHostEnvironment env, Arguments args, IDataView input)
: base(env, RegistrationName, Contracts.CheckRef(args, nameof(args)).Column, input, RowCursorUtils.TestGetLabelGetter)
{
Expand Down
24 changes: 23 additions & 1 deletion src/Microsoft.ML.Data/Transforms/LabelIndicatorTransform.cs
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,18 @@ public bool TryUnparse(StringBuilder sb)
}
}

private static class Defaults
{
public const int ClassIndex = 0;
}

public sealed class Arguments : TransformInputBase
{
[Argument(ArgumentType.Multiple, HelpText = "New column definition(s) (optional form: name:src)", ShortName = "col", SortOrder = 1)]
public Column[] Column;

[Argument(ArgumentType.AtMostOnce, HelpText = "Label of the positive class.", ShortName = "index")]
public int ClassIndex;
public int ClassIndex = Defaults.ClassIndex;
}

public static LabelIndicatorTransform Create(IHostEnvironment env,
Expand Down Expand Up @@ -111,6 +116,23 @@ private static string TestIsMulticlassLabel(ColumnType type)
return $"Label column type is not supported for binary remapping: {type}. Supported types: key, float, double.";
}

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="name">Name of the output column.</param>
/// <param name="source">Name of the input column. If this is null '<paramref name="name"/>' will be used.</param>
/// <param name="classIndex">Label of the positive class.</param>
public LabelIndicatorTransform(IHostEnvironment env,
IDataView input,
string name,
string source = null,
int classIndex = Defaults.ClassIndex)

@TomFinley TomFinley Jul 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This classIndex ought to have been a required parameter. #Closed

@zeahmed zeahmed Jul 5, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that zero is being used as default value. Do you suggest making it required? #Resolved

@TomFinley TomFinley Jul 5, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. So, in these so-called Arguments classes things have default values just because it's easier to have them have defaults, than to make them required, just by the nature of how the structure is used in our existing tooling -- that is, there was a cultural pressure to make the default construction with no arguments work, actually primarily due to the GUI because people thought it was odd that you could add a constructor via the GUI and have it immediately yield an error. That is not a concern here in this case, considering what this is used for, and making it default makes absolutely no sense. This really is something a user should be explicit about.

I do not suggest changing arguments (though you could if you want and I think it would be a great improvement, I think it's really silly that it doesn't require usage to be explicit) but I would certainly require it in the helper constructor. There's nothing fundamentally privileged about the first value... #Closed

: this(env, new Arguments() { Column = new[] { new Column() { Source = source ?? name, Name = name } }, ClassIndex = classIndex }, input)
{
}

public LabelIndicatorTransform(IHostEnvironment env, Arguments args, IDataView input)
: base(env, LoadName, Contracts.CheckRef(args, nameof(args)).Column,
input, TestIsMulticlassLabel)
Expand Down
13 changes: 13 additions & 0 deletions src/Microsoft.ML.Data/Transforms/RangeFilter.cs
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,19 @@ private static VersionInfo GetVersionInfo()
private readonly bool _includeMin;
private readonly bool _includeMax;

/// <summary>
/// Convenience constructor for public facing API.
/// </summary>
/// <param name="env">Host Environment.</param>
/// <param name="input">Input <see cref="IDataView"/>. This is the output from previous transform or loader.</param>
/// <param name="column">Name of the input column.</param>
/// <param name="minimum">Minimum value (0 to 1 for key types).</param>
/// <param name="maximum">Maximum value (0 to 1 for key types).</param>
public RangeFilter(IHostEnvironment env, IDataView input, string column, Double? minimum = null, Double? maximum = null)
: this(env, new Arguments() { Column = column, Min = minimum, Max = maximum }, input)
{
}

public RangeFilter(IHostEnvironment env, Arguments args, IDataView input)
: base(env, RegistrationName, input)
{
Expand Down
Loading