Add @subset

[pdeffebach]

This is an initial attempt to add @subset.

I was able to entirely replace @where by calling skipmissing=true as a keyword argument, which is great.

We are in a real bind with regards to keyword arguments. With the move to using :block we can't just support keyword argument handling like @subset(df, ...; skipmissing = true). So I added the flag @skipmissing and re-factored the macro-flags a little bit.

julia> df = DataFrame(a = [1, missing], b = [3, 4]);

julia> @subset df @skipmissing begin 
           :a .== 1
           :b .== 3
       end
1×2 DataFrame
 Row │ a       b     
     │ Int64?  Int64 
─────┼───────────────
   1 │      1      3

But adding tests is still a pain, because the tests have missing so we would have to add @skipmissing everywhere. Is this a time when we should break with the DataFrames API and make skipmissing=true the default? That would make people's lives a bit easier when upgrading from @where to @subset.

@nalimilan This is a design decision, so I would appreciate your input.

[pdeffebach]

Okay, I've made skipmissing=true the default with subset and removed the @skipmissing flag. This seems like the easiest way forward. Unlike DataFrames.jl, we will treat missing as false in @subset.

All docs are added and tests added and pass. So this could be merged.

[nalimilan]

Thanks! That makes the API much more consistent with DataFrames.

Regarding skipmissing, I think it would be good to outline a general plan. Should DataFramesMeta automatically skip/propagate missing values everywhere? We discussed adding a keyword argument to do that in DataFrames at JuliaData/DataFrames.jl#2314. It hasn't been implemented at this point, but it would make sense to decide whether we would like to enable it by default eventually in DataFramesMeta.

[nalimilan]

Why remove the contents of the docstring?

[pdeffebach]

I will add a correct docstring.

[nalimilan]

Move this to deprecated.jl?

[pdeffebach]

moved.

[nalimilan]

Can you add tests with GroupedDataFrame?

[pdeffebach]

added, ported from @where.

[pdeffebach]

w.r.t. missings.

I think that adding transform! with SubDataFrame goes along way to emulating Stata's if syntax. But you are right it doesn't help with missings.

I think something along the lines of This PR in Missings.jl is the solution. Since we are constructing anonymous functions we can just add a spreadmissing(anon) when we need to. I hope we can make it performant. I don't know if it should be default in case people compare the speed to data.table, but I am open to the idea. It may also help people who don't like row-wise since a lot of the benefit of row-wise functions is dealing with missings.

That's a long term strategy. Maybe in the meantime we should just continue to treat missings as false since it's the default behavior with @where currently.

[nalimilan]

Also test @subset! with GroupedDataFrame?

[pdeffebach]

Added!

[pdeffebach]

Okay ready for merging.

[pdeffebach]

Thank you!