model.kws.kws failed on JSONReader: Unknown field global_key, candidates are:

[insop]

Hi octoml team

Running this command failed on JSONReader: Unknown field global_key, candidates are:

$ python3 -m model.kws.kws --test 1000 --module model/kws/saved/module_gs_4.0.pickle --debug

Error messages are:

$ python3 -m model.kws.kws --test 1000 --module model/kws/saved/module_gs_4.0.pickle --debug

Traceback (most recent call last):
  File "python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "Github/azsphere/python/model/kws/kws.py", line 335, in <module>
    test_accuracy(OPTS, target='llvm --system-lib')
  File "Github/azsphere/python/model/kws/kws.py", line 230, in test_accuracy
    mod = pickle.load(handle)
  File "Github/tvm-upstream/python/tvm/runtime/object.py", line 88, in __setstate__
    self.__init_handle_by_constructor__(_ffi_node_api.LoadJSON, handle)
  File "Github/tvm-upstream/python/tvm/_ffi/_ctypes/object.py", line 131, in __init_handle_by_constructor__
    handle = __init_by_constructor__(fconstructor, args)
  File "Github/tvm-upstream/python/tvm/_ffi/_ctypes/packed_func.py", line 260, in __init_handle_by_constructor__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (8) 9   libtvm.dylib                        0x000000010fec6206 TVMFuncCall + 70
  [bt] (7) 8   libtvm.dylib                        0x000000010f2e1968 void tvm::runtime::TypedPackedFunc<tvm::runtime::ObjectRef (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>::AssignTypedLambda<tvm::runtime::ObjectRef (*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>(tvm::runtime::ObjectRef (*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >))::'lambda'(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const + 472
  [bt] (6) 7   libtvm.dylib                        0x000000010f2d0fbe tvm::LoadJSON(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) + 222
  [bt] (5) 6   libtvm.dylib                        0x000000010f2d1f0e tvm::JSONGraph::Load(dmlc::JSONReader*) + 318
  [bt] (4) 5   libtvm.dylib                        0x000000010f2d9f2e dmlc::JSONObjectReadHelper::ReadAllFields(dmlc::JSONReader*) + 318
  [bt] (3) 4   libtvm.dylib                        0x000000010f2dad7e dmlc::json::ArrayHandler<std::__1::vector<tvm::JSONNode, std::__1::allocator<tvm::JSONNode> > >::Read(dmlc::JSONReader*, std::__1::vector<tvm::JSONNode, std::__1::allocator<tvm::JSONNode> >*) + 238
  [bt] (2) 3   libtvm.dylib                        0x000000010f2db346 tvm::JSONNode::Load(dmlc::JSONReader*) + 566
  [bt] (1) 2   libtvm.dylib                        0x000000010f2da1fc dmlc::JSONObjectReadHelper::ReadAllFields(dmlc::JSONReader*) + 1036
  [bt] (0) 1   libtvm.dylib                        0x000000010f02e5af dmlc::LogMessageFatal::~LogMessageFatal() + 111
  File "Github/tvm-upstream/3rdparty/dmlc-core/include/dmlc/json.h", line 947
JSONReader: Unknown field global_key, candidates are: 
"attrs"
"data"
"keys"
"repr_b64"
"repr_str"
"type_key"

[Bogumil-Sapinski-Mobica]

Hi octoml team,
same issue as @insop described when invoking
make kws
or otherwise
python3 python/build_model.py -o build --keyword --tuned python/tuning/kws/kws_conv_notquantized_runtime.txt --module python/model/kws/saved/module_gs_4.0_conv_notquantized.pickle

Please help.

[Bogumil-Sapinski-Mobica]

Hi octoml team

Running this command failed on JSONReader: Unknown field global_key, candidates are:

$ python3 -m model.kws.kws --test 1000 --module model/kws/saved/module_gs_4.0.pickle --debug

Error messages are:

$ python3 -m model.kws.kws --test 1000 --module model/kws/saved/module_gs_4.0.pickle --debug

Traceback (most recent call last):
  File "python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "Github/azsphere/python/model/kws/kws.py", line 335, in <module>
    test_accuracy(OPTS, target='llvm --system-lib')
  File "Github/azsphere/python/model/kws/kws.py", line 230, in test_accuracy
    mod = pickle.load(handle)
  File "Github/tvm-upstream/python/tvm/runtime/object.py", line 88, in __setstate__
    self.__init_handle_by_constructor__(_ffi_node_api.LoadJSON, handle)
  File "Github/tvm-upstream/python/tvm/_ffi/_ctypes/object.py", line 131, in __init_handle_by_constructor__
    handle = __init_by_constructor__(fconstructor, args)
  File "Github/tvm-upstream/python/tvm/_ffi/_ctypes/packed_func.py", line 260, in __init_handle_by_constructor__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (8) 9   libtvm.dylib                        0x000000010fec6206 TVMFuncCall + 70
  [bt] (7) 8   libtvm.dylib                        0x000000010f2e1968 void tvm::runtime::TypedPackedFunc<tvm::runtime::ObjectRef (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>::AssignTypedLambda<tvm::runtime::ObjectRef (*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)>(tvm::runtime::ObjectRef (*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >))::'lambda'(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const + 472
  [bt] (6) 7   libtvm.dylib                        0x000000010f2d0fbe tvm::LoadJSON(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) + 222
  [bt] (5) 6   libtvm.dylib                        0x000000010f2d1f0e tvm::JSONGraph::Load(dmlc::JSONReader*) + 318
  [bt] (4) 5   libtvm.dylib                        0x000000010f2d9f2e dmlc::JSONObjectReadHelper::ReadAllFields(dmlc::JSONReader*) + 318
  [bt] (3) 4   libtvm.dylib                        0x000000010f2dad7e dmlc::json::ArrayHandler<std::__1::vector<tvm::JSONNode, std::__1::allocator<tvm::JSONNode> > >::Read(dmlc::JSONReader*, std::__1::vector<tvm::JSONNode, std::__1::allocator<tvm::JSONNode> >*) + 238
  [bt] (2) 3   libtvm.dylib                        0x000000010f2db346 tvm::JSONNode::Load(dmlc::JSONReader*) + 566
  [bt] (1) 2   libtvm.dylib                        0x000000010f2da1fc dmlc::JSONObjectReadHelper::ReadAllFields(dmlc::JSONReader*) + 1036
  [bt] (0) 1   libtvm.dylib                        0x000000010f02e5af dmlc::LogMessageFatal::~LogMessageFatal() + 111
  File "Github/tvm-upstream/3rdparty/dmlc-core/include/dmlc/json.h", line 947
JSONReader: Unknown field global_key, candidates are: 
"attrs"
"data"
"keys"
"repr_b64"
"repr_str"
"type_key"

Hi insop,
as there is no reply form octoml team than mayby I will be helpful.
It showed up that it really matters which llvm do you use and tvm.
For me LLVM-10 and TVM git commit f5b02fdb1b5a7b6be79df97035ec1c3b80e3c665 works.
Also when building TVM with LLVM I have setup
set(USE_LLVM "/usr/bin/llvm-config-10 --ignore-libllvm")
in build/config.cmake - there were some linking issues with dynamic libs.

I have switched to Azure Sphere 21.01 - 8+Beta2101. After minor changes it also works with Azure CLI-2.
AzureSphere commands have chaged its style, azsphere_connect.sh moved to other folder - Makefile to review.

However there is also an issue with multiple object definition in intercore.h:6 -> IntercoreComm icc;
This should be declared as "extern".
This leads to linking issue. I do not know if it was build or how?

Leaving this question for maintaners ;)

Best Regards.