ARROW-9761: [C/C++] Add experimental C stream inferface

[pitrou]

The goal is to have a standardized ABI to communicate streams of homogeneous arrays or record batches (for example for database result sets).

The trickiest part is error reporting. This proposal tries to strike a compromise between simplicity (an integer error code mapping to errno values) and expressivity (an optional description string for application-specific and context-specific details).

[pitrou]

cc @wesm

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9761

[emkornfield]

silly question is errno-compatible a well defined unix/windows term? someplace where I can read more about it?

[pitrou]

I don't know how to phrase it: it returns value that are errno error codes (in case of error). A number of values are standard in C++: https://en.cppreference.com/w/cpp/error/errno_macros

[emkornfield]

nit ntew line here or remove the new line above?

[emkornfield]

are arrays produced by this stream tied to the lifecycle of this stream (must they be released first?)

[pitrou]

Hmm... I'd say no.

[emkornfield]

released is something defined in the ArrowArray-spec. Is it stronger or weaker guarantee then returning a nullptr here?

[pitrou]

A nullptr cannot be returned. The callback returns an int. However, we could say that returning -1 means end of stream.

[hannes]

I have one suggestion: The proposal indicates that the stream is finished when the array resulting from get_next is released. This seems a bit odd, how about just setting its length to 0? Or is it possible in the stream API that individual chunks are of length 0 but subsequent chunks are not?

[pitrou]

It's possible to have some chunks with 0 length in a stream, yes. I don't see any reason to forbid it in this API (and such corner cases are often annoying to deal with).

[pitrou]

However, as said above, perhaps returning -1 (rather than 0 for success or a positive errno-like value for errors) could mean an end of stream. Thoughts?

[wesm]

This looks good to me. The next steps would be to integrate this in pyarrow and do the plumbing to get something working end-to-end with a third party project (e.g. DuckDB)

[pitrou]

I added some docs, please take a look.

[pitrou]

Rebased. Does someone want to review the doc at https://pitrou.net/arrowdevdoc/format/CStreamInterface.html ?

[emkornfield]

Rebased. Does someone want to review the doc at https://pitrou.net/arrowdevdoc/format/CStreamInterface.html ?

I took a quick look and seemed reasonable (nothing jumped out at me that needed changes).

[pitrou]

Rebased again. We can probably merge soon, if CI agrees.

[hannes]

One thing we noticed: There is no way to rewind such a stream, but we would really like to be able to do so. For example, if someone defines an Arrow-backed SQL VIEW, it can be scanned multiple times. Once it has been read once, it is currently invalidated and cannot be used again. Would it be possible to add a callback for that? If not supported, it could be set to NULL.

[pitrou]

Would rewind go back to the start of stream always?

[hannes]

Would rewind go back to the start of stream always?

I would think so, no?

[pitrou]

Ok, I just wanted to make sure you weren't looking to go back a number of rows given as argument :-)

[emkornfield]

I think JDBC might in some cases allow for "scrolling" type of rewind vs restart (not saying this should support it but it is worth mentioning.

[pitrou]

Each environment (Python, C++, JDBC, etc.) has its own feature set for streams / iterators, which doesn't make our task easy :-/

[emkornfield]

Starting simple sounds good to me. At least for JDBC, I don't think it is required for driver implementations to support scrolling.

[bkietz]

Rewinding doesn't strike me as something which needs to be part of the C stream protocol. APIs can still provide rewind and other semantics while using a simple-as-possible stream as a building block.

In the case of a SQL view which can be viewed multiple times for example, let the SQL view need not be a stream. It could instead be a function returning streams (each beginning at the start of the view):

-ArrowArrayStream* MakeSqlView(...);
+SqlView* MakeSqlView(...);
+ArrowArrayStream* GetStream(SqlView*);

Rewind, scrolling, offsets, reverse iteration, etc can all be accommodated in this fashion so IMHO they don't belong in the protocol.

[wesm]

Another thing that occurred to me is whether we want to enable batch-level metadata (which would be implementation-defined). This is supported in Flight for example

https://github.com/apache/arrow/blob/master/format/Flight.proto#L316

[pitrou]

I don't think it makes sense to continue hesitating on the API. I think I'm going to merge the PR as-is. The interface is marked experimental so can still evolve (or even be removed).