Tensor Creation from data

[TratsiakY]

Hi.
I've exported my classificator to .onnx and have checked it in python with onnxruntime. All works perfect.
So, I am trying to run my model in c++ now but, unfortunatelly, I haven't got an enought experience in c++. I've studied examples and try to write my own code, but the moment with transform of vector to tensor is not clear for me.
in the case of vector {m}, where m is a number of elements the code below pass the compilation

  vector<int64_t> input_node_dims = { 0, 2 };
  vector<vector<float>> init_data = { { 1.5, 30.0 } }; //toy datafor test
  auto memory_info = MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
  Value input_tensor = Value::CreateTensor<float>(memory_info, init_data.data(), init_data.size(), input_node_dims.data(), 2);
  assert(input_tensor.IsTensor());

(meanwhile, I am not sure that used dims for init_data is correct, but I am not about that).

The main problem is:
How should be written the CreateTensor() in the case of vector{m, n}, where m - rows, n - cols (for example for, vector<vector> init_data = {{a1, b1}, {a, b} })?

[pranavsharma]

You can do vector<float> init_data{1.5, 30.0, 2.0, 3.0, 4.0, 5.0}; for 2 rows and 3 cols (row major format) and then pass it to CreateTensor as init_data.data(). You can also take a look at csharp\test\Microsoft.ML.OnnxRuntime.EndToEndTests.Capi\CXX_Api_Sample.cpp for examples.

[TratsiakY]

You can do vector<float> init_data{1.5, 30.0, 2.0, 3.0, 4.0, 5.0}; for 2 rows and 3 cols (row major format) and then pass it to CreateTensor as init_data.data(). You can also take a look at csharp\test\Microsoft.ML.OnnxRuntime.EndToEndTests.Capi\CXX_Api_Sample.cpp for examples.

@pranavsharma Thank you. I am using one of the examples as template.

in

inline Value Value::CreateTensor(const OrtMemoryInfo* info, T* p_data, size_t p_data_element_count, const int64_t* shape, size_t shape_len)

what is the difference between const int64_t* shape, size_t shape_len?
In https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/CXX_Api_Sample.cpp the const int64_t* shape correspond to the vector which is collect data about structure (1,3,244,244), while 4 is a vector size ()? Or what does mean 4 in this case? Why should we also send the vector size if its size can be calculated automatically inside the CreateTensor?

[TratsiakY]

How can I check the data in created Tensor ?

[pranavsharma]

4 is the number of elements (dimensions) in the shape array pointed to by the 'shape' ptr. Given that CreateTensor is a C API and accepts just a ptr to the shape, it has no idea how many elements (dimensions) the shape array contains. This is why it accepts shape_len as well.
You can use GetTensorMutableData to get a ptr to the data contained in the tensor.

[TratsiakY]

@pranavsharma, Thank you.
A few comments.

My model has input shape {1, 2}.
I creates tensor with corresponding code:

Value input_tensor = Value::CreateTensor(memory_info, inp_data.data(), mdp.TensorSize[i], mdp.DimShape[i].data(), mdp.DimShape[i].size());

where inp_data is array {a, b} (a and b are float, not 2D), mdp.TensorSize[i] = 2;
mdp.DimShape[i] = {1, 2}, mdp.DimShape[i].size() = 2.

on the next step I am checking the input_tensor shape with respect to the model input:

assert(input_tensor.IsTensor() && input_tensor.GetTensorTypeAndShapeInfo().GetShape() == mdp.DimShape[i]);
cout << "\ninput_tensor shape: " << input_tensor.GetTensorTypeAndShapeInfo().GetShape()[0] << input_tensor.GetTensorTypeAndShapeInfo().GetShape()[1] << endl;

output is:

1 2 //As I understand, it is corresponding to the {{a, b}}

on the next step I am converting the Tensor to the vector:

float* floatarr = input_tensor.GetTensorMutableData();

floatarr shape is {0, 2}. It is match with inp_data.

finally, I am trying to run the model:

auto output_tensors = ses.Run(RunOptions{ nullptr }, mdp.InputLayerName.data(), &input_tensor, mdp.Ninp, mdp.OututLayerName.data(), mdp.OututLayerName.size());

where mdp.InputLayerName.data() - array with input names {"data_in"}; mdp.Ninp - number of inputs (1); mdp.OututLayerName.data() - array with output names {"label", "label_prob"}; mdp.OututLayerName.size() - number of outputs (2);

The code passes the compilation but throw Exception related with access to the memory when run.

Вызвано исключение по адресу 0x00007FFEF1E5A839 в ClassificatorCaller.exe: исключение Microsoft C++: std::logic_error по адресу памяти 0x0000009B356FBDA0.

Debugger refers to this line

ThrowOnError(Global::api_.Run(p_, run_options, input_names, ort_input_values, input_count, output_names, output_count, ort_output_values));

I suppose that main problem can be related with both: tensor structure or creating output tensor. Or I am doing in general something wrong?

[jgbradley1]

@TratsiakY I have submitted a PR here that will make it easier to check data in a tensor. I had the same challenges as you. It will allow you to access tensor data similar to how OpenCV Mat's At() functionality works (if you're familiar with that library) by passing in a specific location of the tensor data blob (i.e. [1,128,250] in a 3-dimensional blob).

Not sure why but in my experience, it does seem to take an exceptional amount of time for PRs to get reviewed and merged into master. @pranavsharma is there someone dedicated to working on the C++ API that I should ping when requesting a review?

[jgbradley1]

@TratsiakY A couple other things that are not very apparent in the C++ documentation.

1. onnxruntime (ORT) tensors assume a row-major order of the data
2. ORT tensors do not copy data. They are only a thin wrapper around a block of data owned by another object (in your example, that would be the std::vector init_data).

If you changed the values of init_data directly after creating the ORT tensor, you would see those changes reflected in the ORT tensor when you tried to access the data via the ORT tensor API.

[pranavsharma]

Looks ok. What status message is returned? I can't follow this "Вызвано исключение по адресу 0x00007FFEF1E5A839 в ClassificatorCaller.exe: исключение Microsoft C++: std::logic_error по адресу памяти 0x0000009B356FBDA0."

[TratsiakY]

@jgbradley1, @pranavsharma thank you. Seems thatwe have a bug, probably .
I used skl2onnx library for convert my model to onnx. skl2onnx create two output layers: label_output (0 or 1 value) and label_probability (type: sequence<map<int64,float32>>). Thus, I was trying to receive both outputs, that was noted in previous post.
The main problem relates with creation of output tensor.

I used the squeezenet example with minor modification for my model (input and output layers quantity and names). The error was the same, as was mentioned before :

The exception on adress 0x00007FFEF1E5A839 was called in MyProject.exe: exception Microsoft C++: std::logic_error on memory address 0x0000009B356FBDA0 (the memory cells addresses for different runs were different)

finally, I tryed to get the prediction only for one output layer. I was surprised, because the code was passed the compilation and worked properly.
the code is listed below:

// Copyright(c) Microsoft Corporation.All rights reserved.
// Licensed under the MIT License.
//
#include <assert.h>
#include
#include <onnxruntime_cxx_api.h>
#include
#include
using namespace std;
int main(int argc, char* argv[]) {
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "test");

Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);

session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);

#ifdef _WIN32
const wchar_t* model_path = L"test.onnx";
#else
const char* model_path = "test.onnx";
#endif
printf("Using Onnxruntime C++ API\n");
Ort::Session session(env, model_path, session_options);

Ort::AllocatorWithDefaultOptions allocator;

size_t num_input_nodes = session.GetInputCount();
std::vector<const char*> input_node_names(num_input_nodes);
std::vector<int64_t> input_node_dims;  // simplify... this model has only 1 input node {1, 3, 224, 224}.

printf("Number of inputs = %zu\n", num_input_nodes);

for (int i = 0; i < num_input_nodes; i++) {
    // print input node names
    char* input_name = session.GetInputName(i, allocator);
    printf("Input %d : name=%s\n", i, input_name);
    input_node_names[i] = input_name;

    // print input node types
    Ort::TypeInfo type_info = session.GetInputTypeInfo(i);
    auto tensor_info = type_info.GetTensorTypeAndShapeInfo();

    ONNXTensorElementDataType type = tensor_info.GetElementType();
    printf("Input %d : type=%d\n", i, type);

    // print input shapes/dims
    input_node_dims = tensor_info.GetShape();
    printf("Input %d : num_dims=%zu\n", i, input_node_dims.size());
    for (int j = 0; j < input_node_dims.size(); j++)
        printf("Input %d : dim %d=%jd\n", i, j, input_node_dims[j]);
}

size_t input_tensor_size = 2;  // simplify ... using known dim values to calculate size
                                           // use OrtGetTensorShapeElementCount() to get official size!

std::vector<float> input_tensor_values(input_tensor_size);
std::vector<const char*> output_node_names = { "output_label"};

// initialize input data with values in [0.0, 1.0]

input_tensor_values = {1.5, 1};

// create input tensor object from data values
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
Ort::Value input_tensor = Ort::Value::CreateTensor<float>(memory_info, input_tensor_values.data(), input_tensor_size, input_node_dims.data(), 2);
assert(input_tensor.IsTensor());

// score model & input tensor, get back output tensor
auto output_tensors = session.Run(Ort::RunOptions{ nullptr }, input_node_names.data(), &input_tensor, 1, output_node_names.data(), 1);
assert(output_tensors.size() == 1 && output_tensors.front().IsTensor());

// Get pointer to output tensor float values
int* floatarr = output_tensors[0].GetTensorMutableData<int>();

cout << floatarr[0] << endl;

printf("Done!\n");
return 0;

}

Meanwhile, the code pass the compilation but fall at run if I try to get the array with predictions output.

The error occurs in place where the output tensor is created:

auto output_tensors = session.Run(Ort::RunOptions{ nullptr }, input_node_names.data(), &input_tensor, 1, output_node_names.data(), 2);

[TratsiakY]

I've found yet one strange moment.
I've formatted my code to a class for exclude the requirence in reading of the onnx file everytime. The class is very simple and consist of only two methods, one of them is constructor, while the second one is predictor.
For correct work I should run the prediction once in constructor with any data, otherwise I will have got the error message related with acces to memory. After that method with prediction works well.

[TratsiakY]

@pranavsharma, @jgbradley1, Please, find attached .cpp files and onnx model for play and repeat the error.
time.zip

[pranavsharma]

@TratsiakY you must ensure the environment instance outlives the session. In your example, the env instance gets destructed at the end of the PredictWithONNXModel constructor. If you move the environment to a class member, it should work just fine.

[Karnav123]

@TratsiakY you must ensure the environment instance outlives the session. In your example, the env instance gets destructed at the end of the PredictWithONNXModel constructor. If you move the environment to a class member, it should work just fine.

Will making env a static variable work? I have a similar issue in which the prediction for the same data for the very first time is not correct but it works fine for second time and further.