Is there anyway I can get the intermediate outputs and nodes information?

[han-minhee]

in onnxruntime, I could get the intermediate outputs and nodes information using onnx.ModelProto,
but I can't find a way to do such using onnxruntime-genai.
Is there any way I can achieve this, or is it simply not possible at the moment?

[kunal-vaishnavi]

If you build ONNX Runtime with onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS and then build ONNX Runtime GenAI normally, you should see all intermediate outputs and node information. You can export the environment variables as needed to get the information you want.

[han-minhee]

Thank you! I haven't confirmed the output data but could get node info. I will let you know after I check it

[han-minhee]

I'm sorry to ask you here more about onnxruntime, but where would be the inputs and outputs saved as files? I tried setting ORT_DEBUG_NODE_IO_OUTPUT_DIR but to no avail.

[kunal-vaishnavi]

You can try exporting multiple environment variables for that.

export ORT_DEBUG_NODE_IO_DUMP_DATA_DESTINATION="files"
export ORT_DEBUG_NODE_IO_DUMPING_DATA_TO_FILES_FOR_ALL_NODES_IS_OK=1
export ORT_DEBUG_NODE_IO_OUTPUT_DIR="output_dir"

Alternatively, you can dump the input and output data

export ORT_DEBUG_NODE_IO_DUMP_INPUT_DATA=1
export ORT_DEBUG_NODE_IO_DUMP_OUTPUT_DATA=1
export ORT_DEBUG_NODE_IO_DUMP_SHAPE_DATA=1

and then redirect the dumped data to a log file. For example:

python run.py > output.log 2>&1

[han-minhee]

I confirmed the tensorproto files are being produced. However, as the inputs or outputs are written to the files with the tensor names as file names, I can't distinguish which is for which node. Most tensors are distinguishable as their names include the name of the nodes they are from, but I guess there might be some cases that isn't so.

Thank you a lot!