[discussion] Support Advanced NAS Algorithms

[gaocegege]

Now we are using a custom IR (architecture + nn_config) to represent a candidate model, which may be insufficiently expressive, I think.

I think we should use mature IRs such as Relay IR or ONNX to represent the candidate model.

/cc @DeeperMind @andreyvelich

[gaocegege]

Ref apache/tvm#1673

[andreyvelich]

@gaocegege This is make sense. What changes in API do we need to make IR like Relay IR or ONNX. And what will be the best for the NAS tasks?
/cc @johnugeorge @richardsliu @hougangliu @YujiOshima

[gaocegege]

We need to investigate which IR works well for the case. microsoft/nni uses ONNX. While I think ONNX is not expressive.

@anchovYu is working on this issue.

[andreyvelich]

That is cool! Just let us know which structure you want to use and will our definition of Neural Network work for that. For RL you can find it here.

[gaocegege]

@andreyvelich Yeah, sure. Maybe we could discuss it in the katib meeting. Actually, we are not sure about the choice, too.

There are several ways to represent the search space of NAS: chain-structured (what we are using now), chain-structured with jumps and other advanced links, and cell-structured. I cannot figure a way to represent such complex space definition, even via ONNX or Relay IR or other IRs.

[andreyvelich]

Alright, let discuss at Katib meeting in this week.
/cc @DeeperMind

[gaocegege]

https://github.com/microsoft/onnxruntime/tree/master/orttraining ONNX runtime supports training ONNX. Maybe we could have a look.

/cc @andreyvelich

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/katib	0.74
kind/question	0.84

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/lifecycle frozen