ARROW-11979: Combine limit into SortOptions - REQUEST FOR COMMENT

[alamb]

The sort_limit kernel was added by @sundy-li in #9602

While writing some doc examples in #9721, it occured to me we could potentially simplify the API so I figured I would offer a proposed PR for comment

Rationale

Since we already have a SortOptions structure that controls sorting options, we could also add the limit to that structure rather than adding a new sort_limit function and still avoid changing the API

Changes

Move the limit option to SortOptions

[alamb]

Here is the proposal

[alamb]

I will rewrite the tests properly (with a single SortOptions parameter) if people like this API change, but I didn't bother until I know it is worthwhile

[alamb]

@Dandandan I wonder if you have any comments on this approach

[jorgecarleitao]

I think that SortOptions is used on a per-column basis; it corresponds to how each column should be ordered (nulls first, descending). limit reflects how many entries should be outputted by sort, which IMO is column-independent.

This is probably more evident in lex_sort, where a SortOption per column is passed.

[alamb]

I think that SortOptions is used on a per-column basis; it corresponds to how each column should be ordered (nulls first, descending). limit reflects how many entries should be outputted by sort, which IMO is column-independent.

@jorgecarleitao that is a good point. Since the Arrow sort kernel only operates on a single column, I think the distinction is not relevant.

However, since lexsort and DataFusion sort use the same SortOptions struct for each coulmn, that is a good reason not to add a limit option to them (as then it could be inconsistent across columns, potentially)

[sundy-li]

I think that SortOptions is used on a per-column basis; it corresponds to how each column should be ordered (nulls first, descending). limit reflects how many entries should be outputted by sort, which IMO is column-independent.

Totally agree with that.

select * from table order by a ASC, b DESC limit 10

There are 2 sort_options, yet the limit is the only one for the global results.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-11979

[alamb]

I am convinced that this PR is not a good idea at this time. Closing. Thanks for the feedback @sundy-li and @jorgecarleitao 👍