Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions format/Flight.proto
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,44 @@ message Location {
*/
message Ticket {
bytes ticket = 1;

// Data representing some part of the data retrievable by the ticket.
//
// This is an optional optimization for applications that want to reduce latency,
// for previewing results or retrieving small data sets by providing data
// without requiring a DoGet call. Applications built on top of Flight
// are responsible for any negotiation necessary on whether
// inlining data is appropriate.
//
// The size of inlined_data is expected to be small (typically less then 1MB)
// and inlining too much data across tickets can run into underlying transport
Copy link

@GavinRay97 GavinRay97 Mar 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any hard-set number to be aware of here? Or some general guidance/recommendations?
Seems important not to accidentally hit payload size limit in the transport

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might already be overriding but gRPC has a default of 4MB limit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hesitant to document this since there are experiments with other transports.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah that is good to know, thank you. Will have to see how that can be configured because there are some cases in which a query may fetch ~10MB of data (or more)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's up to the transport. I believe we override this by default in C++ and Java, yeah, but I don't think we should document it here.

// limitations. Furthermore, since the data is expected to be small, implementations
// are less likely to optimize for zero-copy in these cases.
message InlinedData {
enum DataSubsetType {
// Indicates server is using a different version
// of the protos.
DATA_SUBSET_TYPE_UNDEFINED = 0;
// The data present in inlined_data represents all data
// present for the ticket.
COMPLETE_DATA = 1;
// The data present in inlined_data represents only a partial
// sample of the data available for the ticket. Applications
// may specify more requirements for partial data (e.g. always
// represents the leading n-rows of data) but are not required
// to do so.
PARTIAL_DATA = 2;
}
DataSubsetType inlined_completeness = 1;

// Arrow data for consumption.
//
// The data is not expected to contain the schema message. The schema
// should be identical to the schema provided on FlightInfo.
repeated FlightData data = 2;
}
InlinedData inlined_data = 2;

}

/*
Expand Down