[TVMC] Allow selecting a subset of tasks to be used in `tvmc tune` #12525

PhilippvK · 2022-08-22T06:48:03Z

This adds a --tasks flag to the tvmc tune command to filter the lists of tasks to be tuned. See examples below.

Motivation

As auto-tuning can be quite time consuming, it is often desirable to cut down the number of tuned tasks in a session.
If the tuning session was canceled halfway through, it would be a bad idea to start from scratch. Instead continue with the last untuned task
Some tasks have more impact on the model performance than others, thus we should be able to train some tasks longer than others

Examples

Use --task list to show which tasks are available for tuning

$ tvmc tune toycar.tflite -o out.txt --task list
Available Tasks for tuning:
  0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), 'int...
  1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), 'int16...
  2. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (128, 128), 'int...
  3. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (128, 128), 'int16...
  4. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (8, 128), 'int16...
  5. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (8, 128), 'int16')...
  6. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 8), 'int16'), ('TENSOR', (128, 8), 'int16')...
  7. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 8), 'int16'), ('TENSOR', (128, 8), 'int16'), ...
  8. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (640, 128), 'int...
  9. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (640, 128), 'int16...

Filter the list of tasks to be tuned:

# Only tune a single task (index 5)
tvmc tune toycar.tflite -o out.txt --tasks 5

# Tunes tasks starting with index 6
tvmc tune toycar.tflite -o out.txt --tasks "6-"

# Tune tasks 1,4,5,6,8,9
tvmc tune toycar.tflite -o out.txt --tasks "1,4-6,8-"

Tests

I added a basic unit test for the filter_tasks utility in tests/python/driver/tvmc/test_autotuner.py.

Open Questions

While the (truncated) string representations of AutoTVM tasks are quite helpful to pick the correct tasks, using AutoScheduler the tasks can not really be distinguished from each other (only by index). Is there a way to get similar information from AutoScheduler tasks?

cc @Mousius @gromero

PhilippvK · 2022-08-22T06:49:35Z

CC @leandron @giuseros @areusch @tqchen

areusch

thanks @PhilippvK, this looks like a great addition! left a couple comments here and there

python/tvm/driver/tvmc/autotuner.py

tests/python/driver/tvmc/test_autotuner.py

areusch · 2022-09-07T20:36:14Z

@PhilippvK i think this one needs a rebase

PhilippvK · 2022-09-09T14:23:23Z

Short update:

With my latest changes I can now print useful task names for AutoScheduler as well (using task.desc instead of str(task))
I added a mention of the --task argument in the TVMC CLI tutorial

areusch

thanks @PhilippvK, could you address the lint issue?

PhilippvK · 2022-09-21T09:09:58Z

@areusch

As I have to retrigger the CI anyway so I want to propose another small addition:

For AutoTVM, in addition to the task description, I could also print len(task.config_space) to give an idea on how large the tuning search space is for a specific task, which can help a lot to estimate the required number of tuning trials.

Example:

Available Tasks for tuning:
  0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640)... (len=640)
  1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), ... (len=35)

Do you think this would be a good addition to tvmc tune --task list?

PhilippvK · 2022-10-14T07:03:32Z

@areusch Can I have an update on this?

leandron · 2022-10-14T08:24:46Z

@tvm-bot rerun

areusch · 2022-10-17T20:23:44Z

sorry @PhilippvK I missed this one. feel free to ping me on discord if that happens again. I retriggered the CI, I'm good for @leandron to merge this if it passes.

Example:

Available Tasks for tuning:
0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640)... (len=640)

Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), ... (len=35)
Do you think this would be a good addition to tvmc tune --task list?

Sure that seems like a good addition!

tvm-bot · 2022-10-18T09:39:46Z

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

Built docs for commit ff33ca0 can be found here.

_{Generated by tvm-bot}

PhilippvK · 2022-10-18T10:32:49Z

Rebased and CI should be fixed now @leandron @areusch.

leandron · 2022-11-19T09:28:57Z

Sorry @PhilippvK. It took me a while to catch up with this. Can you rebase and then we move to get this merged?

PhilippvK · 2022-12-22T16:01:51Z

@leandron Rebased on upstream/main

areusch · 2023-01-19T19:29:55Z

@tvm-bot rerun

PhilippvK · 2023-02-09T16:00:06Z

Rebased ontop of main and CI errors seem to be fixed. Can we merge this @areusch @leandron ?

PhilippvK · 2023-03-02T15:59:31Z

Any further comments @areusch @leandron?

PhilippvK · 2023-03-21T19:10:45Z

Rebased to fix conflict + I added 2 small AutoTVM related improvements:

If #trials is less then, #tasks, tune each task at least once, instead of failing all of them.
If the config space of a single task is smaller than the determined number of trials per task, use the smaller values as bound in the progress bar.

[TVMC] Address PR comments [TVMC] Properly print task names of for AutoScheduler [TVMC] Implement print_task_list to cleanup code

[Tests] add missing tvm.testing.main call to test_autotuner.py

…e_driver.py

lhutton1

I just noticed this PR and think it looks very helpful, thanks! I had a couple of nits around the testing, but given this PR has been rebased a few times I'm happy for them to be fixed in a follow-up, let me know :)

tests/python/driver/tvmc/test_autotuner.py

lhutton1 · 2023-03-28T21:19:38Z

tests/python/driver/tvmc/test_autotuner.py

    assert mock_pc.call_args_list[0][1]["opt_level"] == 3
+
+
+def test_filter_tasks_valid():


Nit: it would be helpful to parameterize these tests similar to below, so failures are reported separately in the CI log

Oops I forgot to address this. Can we merge this anyway?

python/tvm/driver/tvmc/autotuner.py

lhutton1 · 2023-03-29T21:25:58Z

@tvm-bot rerun

PhilippvK · 2023-03-30T06:38:43Z

@lhutton1 Hey, I implemented the proposed changes and the CI passed. Feel free to merge after quickly having a look at the refactored unit tests.

lhutton1

Thanks @PhilippvK, LGTM!

lhutton1 · 2023-03-30T13:13:40Z

Thanks @PhilippvK @areusch @leandron!

PhilippvK force-pushed the feature_tvmc_tune_tasks branch from d249a35 to 7085107 Compare August 22, 2022 06:48

areusch reviewed Aug 25, 2022

View reviewed changes

github-actions bot requested review from Mousius and gromero September 7, 2022 20:15

PhilippvK force-pushed the feature_tvmc_tune_tasks branch 2 times, most recently from e4d85de to c1306f7 Compare September 9, 2022 13:04

areusch approved these changes Sep 15, 2022

View reviewed changes

PhilippvK force-pushed the feature_tvmc_tune_tasks branch from c4f6894 to 1e7d121 Compare October 18, 2022 07:29

areusch added needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it and removed needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it labels Oct 19, 2022

PhilippvK force-pushed the feature_tvmc_tune_tasks branch 4 times, most recently from c66a3e6 to 6c9c16b Compare November 22, 2022 13:05

PhilippvK force-pushed the feature_tvmc_tune_tasks branch from 6c9c16b to ff33ca0 Compare November 30, 2022 10:59

PhilippvK force-pushed the feature_tvmc_tune_tasks branch from ff33ca0 to 631f97d Compare December 22, 2022 13:02

PhilippvK force-pushed the feature_tvmc_tune_tasks branch from 631f97d to f11bb22 Compare February 9, 2023 08:05

PhilippvK force-pushed the feature_tvmc_tune_tasks branch from f11bb22 to 1aff401 Compare March 21, 2023 19:08

PhilippvK and others added 7 commits March 22, 2023 11:01

[TVMC] Allow section of tasks to be tuned when using tvmc tune command

6bf88fe

[TVMC] Address PR comments [TVMC] Properly print task names of for AutoScheduler [TVMC] Implement print_task_list to cleanup code

[Tests] [TVMC] add tests for filter_tasks utility

942d2f7

[Tests] add missing tvm.testing.main call to test_autotuner.py

[TVM][Tutorial] mention --task flag for tvmc tune in tvmc_command_lin…

1b48dcc

…e_driver.py

[TVMC] Lint filter_tasks code

ffc153b

[TVMC] Add size of task space to tvmc tune --tasks output

8b60b7e

[TVMC] Tune each AutoTVM tasks at least once

bd94217

[TVMC] Use len of AutoTVM config space in progress bar

858a722

PhilippvK force-pushed the feature_tvmc_tune_tasks branch from 1aff401 to 858a722 Compare March 22, 2023 10:01

lhutton1 approved these changes Mar 28, 2023

View reviewed changes

PhilippvK added 3 commits March 29, 2023 16:45

[TVMC] improve assertions in filter_tasks

f7fd11a

[TVMC] refactor print_task_list into gen_task_list

02c5b1e

[TVMC] improve unit tests

dee6607

add missing docstring

a1ab92a

lhutton1 approved these changes Mar 30, 2023

View reviewed changes

lhutton1 merged commit ffc1fc0 into apache:main Mar 30, 2023

ysh329 mentioned this pull request Apr 17, 2023

[Release] v0.12.0 Release Candidate Notes #14645

Closed

PhilippvK deleted the feature_tvmc_tune_tasks branch January 23, 2024 17:07

		assert mock_pc.call_args_list[0][1]["opt_level"] == 3


		def test_filter_tasks_valid():

[TVMC] Allow selecting a subset of tasks to be used in tvmc tune #12525

[TVMC] Allow selecting a subset of tasks to be used in tvmc tune #12525

Uh oh!

Conversation

PhilippvK commented Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Examples

Tests

Open Questions

Uh oh!

PhilippvK commented Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

areusch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

areusch commented Sep 7, 2022

Uh oh!

PhilippvK commented Sep 9, 2022

Uh oh!

areusch left a comment

Choose a reason for hiding this comment

Uh oh!

PhilippvK commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PhilippvK commented Oct 14, 2022

Uh oh!

leandron commented Oct 14, 2022

Uh oh!

areusch commented Oct 17, 2022

Uh oh!

tvm-bot commented Oct 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PhilippvK commented Oct 18, 2022

Uh oh!

leandron commented Nov 19, 2022

Uh oh!

PhilippvK commented Dec 22, 2022

Uh oh!

areusch commented Jan 19, 2023

Uh oh!

PhilippvK commented Feb 9, 2023

Uh oh!

PhilippvK commented Mar 2, 2023

Uh oh!

PhilippvK commented Mar 21, 2023

Uh oh!

lhutton1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lhutton1 Mar 28, 2023

Choose a reason for hiding this comment

Uh oh!

PhilippvK Mar 30, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lhutton1 commented Mar 29, 2023

Uh oh!

PhilippvK commented Mar 30, 2023

Uh oh!

lhutton1 left a comment

Choose a reason for hiding this comment

Uh oh!

lhutton1 commented Mar 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

[TVMC] Allow selecting a subset of tasks to be used in `tvmc tune` #12525

[TVMC] Allow selecting a subset of tasks to be used in `tvmc tune` #12525

PhilippvK commented Aug 22, 2022 •

edited

Loading

PhilippvK commented Aug 22, 2022 •

edited

Loading

PhilippvK commented Sep 21, 2022 •

edited

Loading

tvm-bot commented Oct 18, 2022 •

edited

Loading