[Enhancement]: Data Exchange Optimization

[lovelynewlife]

Background

Since the structure and memory layout of intermediate data required by OB vanilla query engine does not match with that required by inference engine e.g. OB datums and numpy ndarray , it's necessary to transform data from database to the inference specified data type. We support OB tuple datums-> numpy data transform as default option for now. However, there are several issues in current implementation which may incur query efficiency degradation:

• Repetitive target data allocation and construction: target data means the arguments data transferred to Python world. Under the **Prediction Operator Context, ** we are able to get a continuous memory and have its full control but the target data objects still need to allocate and construct for each query iteration.
• Inefficient string data exchange: string transformation is tricky. Maybe due to the differences in memory layouts or encoding methods. Calling different inference engines may incur redundant string copy. e.g. calling ONNXRuntime using numpy string ndarray have two string transformation processes.

Flame Graph Reference

Using Sklearn

Using ONNXRuntime

Solution for each Issue

• Construct target data directly on the buffer memory.
• A unified string intermediate data view for string exchange.