Enhancing TableVectorizer API

[Vincent-Maladiere]

Basic use-case

Last week, @GaelVaroquaux mentioned the need to add more hyper-parameters to the TableVectorizer and be able to select encoders for specific columns.

After discussing the new API design with @ogrisel and @glemaitre, one solution could be to add arbitrary keywords directly to the init:

tv = TableVectorizer(
    numerical_transformer=StandardScaler(),
    high_card_cat_transformer=MinHashEncoder(),
    temperature_index=RobustScaler(),
)

Here temperature_index is a numerical column, and since we pass it to TableVectorizer as a keyword argument, we overwrite StandardScaler with RobustScaler instead.

Advanced use-case

Additionally, to support:

1. column as indexes for numpy arrays, like [1, 3, 3]
2. column with non-alpha-numeric characters or beginning with a number, like "0temperature"

we could replicate the logic of ColumnTransformer with a list of tuples:

tv = TableVectorizer(
    numerical_transformer=StandardScaler(),
    high_card_cat_transformer=MinHashEncoder(),
    column_specific_transformers=[
        ("rs", RobustScaler(), "0temperature"),
    ],
)

Advantages

• Simpler and more user-friendly than having to define a list of tuples like ColumnTransformer right away.

Limitations

• Having both options of using kwargs and column_specific_transformers might be confusing
• Rarely, this could lead to some keyword name-clash

What do you all think of this?

cc @LilianBoulard

[LilianBoulard]

Hi, thanks for the post!

I like the column_specific_transformers argument, I think this is the best solution.

Having both options of using kwargs and column_specific_transformers might be confusing

I strongly agree, and I notice another issue from the user point of view: when I see the call

tv = TableVectorizer(
    numerical_transformer=StandardScaler(),
    high_card_cat_transformer=MinHashEncoder(),
    temperature_index=RobustScaler(),
)

Without reading the TableVectorizer's doc, it's not clear which arguments are column names and which are not. Furthermore, column names might be illegal for Python arguments (e.g. 0day_description, Job title). So even if somebody was all-in on the kwargs syntax, sometimes they would be forced to use column_specific_transformers, thus having to use the two syntaxes (confusing), or having to move their previous arguments to the tuple format (annoying).

I absolutely agree kwargs are cool, but I don't think that's an appropriate use-case :)

Rarely, this could lead to some keyword name-clash

Interesting remark, though it's a problem things like the Pipeline or the ColumnTransformer surely know how to handle, and we could re-use their methodology.

[GaelVaroquaux]

I like the column_specific_transformers argument, I think this is the best solution. Having both options of using kwargs and column_specific_transformers might be confusing I strongly agree, and I notice another issue from the user point of view: when I see the call

I agree with what Lilian points out and I share his views. Thanks to both of you!

[jovan-stojanovic]

I agree, the most elegant solution for me is with the column_specific_transformers!

[Vincent-Maladiere]

#583 closes this discussion