refactor dialogue state tracking for modelling/dataset interoperability #3526

Zhilin123 · 2022-01-26T23:11:07Z

Signed-off-by: Zhilin Wang [email protected]

Design based on attached diagram
UML proposed dialogue1.pdf

Currently supports SGD-QA (Bert-based) and Huggingface GPT2 models, and planning to support other models.

Signed-off-by: Zhilin Wang <[email protected]>

Zhilin123 · 2022-01-26T23:21:03Z

Integration testing done by comparing output of 1. modified Dataset/DataProcessor with original SGDQA model and 2. original Dataset/DataProcessor with original SGDQA model. Both give identical output

CUDA_VISIBLE_DEVICES=0 python ~/NeMo/examples/nlp/dialogue_state_tracking_generative/sgd_gen.py do_training=False trainer.gpus=1 model.dataset.data_dir=/home/zhilinw/dstc8-schema-guided-dialogue/ model.dataset.dialogues_example_dir=/home/zhilinw/dstc8-schema-guided-dialogue/ model.dataset.use_cache=False

and

CUDA_VISIBLE_DEVICES=0 python ~/NeMo/examples/nlp/dialogue_state_tracking/sgd_qa.py do_training=False trainer.gpus=1 model.dataset.data_dir=/home/zhilinw/dstc8-schema-guided-dialogue/ model.dataset.dialogues_example_dir=/home/zhilinw/dstc8-schema-guided-dialogue/ model.dataset.use_cache=False

[NeMo I 2022-01-25 16:32:03 evaluate:284] Dialog metrics for #SEEN_SERVICES : [('active_intent_accuracy', 38.36), ('average_cat_accuracy', 13.69), ('average_cat_status_accuracy', 100.0), ('average_cat_value_accuracy', 13.69), ('average_goal_accuracy', 25.8), ('average_noncat_accuracy', 30.83), ('average_noncat_status_accuracy', 98.83), ('average_noncat_value_accuracy', 31.19), ('joint_cat_accuracy', 4.19), ('joint_cat_status_accuracy', 47.69), ('joint_cat_value_accuracy', 9.39), ('joint_goal_accuracy', 0.25), ('joint_noncat_accuracy', 4.59), ('joint_noncat_status_accuracy', 22.48), ('joint_noncat_value_accuracy', 13.44), ('requested_slots_f1', 2.7), ('requested_slots_precision', 3.05), ('requested_slots_recall', 90.75)] [NeMo I 2022-01-25 16:32:03 evaluate:286] Dialog metrics for #UNSEEN_SERVICES: [('active_intent_accuracy', 33.82), ('average_cat_accuracy', 18.37), ('average_cat_status_accuracy', 100.0), ('average_cat_value_accuracy', 18.37), ('average_goal_accuracy', 25.22), ('average_noncat_accuracy', 27.62), ('average_noncat_status_accuracy', 97.85), ('average_noncat_value_accuracy', 28.22), ('joint_cat_accuracy', 1.18), ('joint_cat_status_accuracy', 25.6), ('joint_cat_value_accuracy', 11.14), ('joint_goal_accuracy', 0.51), ('joint_noncat_accuracy', 2.22), ('joint_noncat_status_accuracy', 42.34), ('joint_noncat_value_accuracy', 9.29), ('requested_slots_f1', 4.21), ('requested_slots_precision', 4.25), ('requested_slots_recall', 91.42)] [NeMo I 2022-01-25 16:32:03 evaluate:288] Dialog metrics for #ALL_SERVICES : [('active_intent_accuracy', 34.85), ('average_cat_accuracy', 17.2), ('average_cat_status_accuracy', 100.0), ('average_cat_value_accuracy', 17.2), ('average_goal_accuracy', 25.35), ('average_noncat_accuracy', 28.36), ('average_noncat_status_accuracy', 98.08), ('average_noncat_value_accuracy', 28.91), ('joint_cat_accuracy', 1.92), ('joint_cat_status_accuracy', 31.04), ('joint_cat_value_accuracy', 10.7), ('joint_goal_accuracy', 0.45), ('joint_noncat_accuracy', 2.76), ('joint_noncat_status_accuracy', 37.84), ('joint_noncat_value_accuracy', 10.25), ('requested_slots_f1', 3.87), ('requested_slots_precision', 3.98), ('requested_slots_recall', 91.27)]

lgtm-com · 2022-01-26T23:21:21Z

This pull request introduces 51 alerts when merging b287422 into 6b51350 - view on LGTM.com

new alerts:

25 for Unused import
16 for Unused local variable
4 for Modification of parameter with default
2 for Superclass attribute shadows subclass method
1 for Unnecessary pass
1 for Except block handles 'BaseException'
1 for Unreachable code
1 for Explicit export is not defined

nemo/collections/nlp/modules/common/gpt_module.py

Signed-off-by: Zhilin Wang <[email protected]>

…/NVIDIA/NeMo into main

lgtm-com · 2022-01-28T00:15:56Z

This pull request introduces 51 alerts when merging 7c12ab6 into 101977e - view on LGTM.com

new alerts:

25 for Unused import
16 for Unused local variable
4 for Modification of parameter with default
2 for Superclass attribute shadows subclass method
1 for Unnecessary pass
1 for Except block handles 'BaseException'
1 for Unreachable code
1 for Explicit export is not defined

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-01-31T22:12:13Z

This pull request introduces 13 alerts when merging 255294d into 9a1cc36 - view on LGTM.com

new alerts:

4 for Unused import
4 for Modification of parameter with default
2 for Superclass attribute shadows subclass method
1 for Unnecessary pass
1 for Unused local variable
1 for Except block handles 'BaseException'

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-01-31T23:02:50Z

This pull request introduces 13 alerts when merging 5f6dbd9 into ddcc2a6 - view on LGTM.com

new alerts:

4 for Unused import
4 for Modification of parameter with default
2 for Superclass attribute shadows subclass method
1 for Unnecessary pass
1 for Unused local variable
1 for Except block handles 'BaseException'

lgtm-com · 2022-01-31T23:36:16Z

This pull request introduces 13 alerts when merging deeeaec into ddcc2a6 - view on LGTM.com

new alerts:

4 for Unused import
4 for Modification of parameter with default
2 for Superclass attribute shadows subclass method
1 for Unnecessary pass
1 for Unused local variable
1 for Except block handles 'BaseException'

okuchaiev · 2022-02-01T01:22:41Z

/blossom-ci

okuchaiev · 2022-02-01T01:23:27Z

ping @cparisien and @yzhang123

cparisien

This looks good to me from a design perspective. We should still get a review from someone who is a regular nemo contributor, since I'm not.

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-02-01T23:22:02Z

This pull request introduces 13 alerts when merging 7d96cf2 into 7a8a5b3 - view on LGTM.com

new alerts:

4 for Unused import
4 for Modification of parameter with default
2 for Superclass attribute shadows subclass method
1 for Unnecessary pass
1 for Unused local variable
1 for Except block handles 'BaseException'

…/NVIDIA/NeMo into main

lgtm-com · 2022-02-02T01:08:55Z

This pull request introduces 13 alerts when merging 0b1bc6c into 7a8a5b3 - view on LGTM.com

new alerts:

4 for Unused import
4 for Modification of parameter with default
2 for Superclass attribute shadows subclass method
1 for Unnecessary pass
1 for Unused local variable
1 for Except block handles 'BaseException'

lgtm-com · 2022-02-02T19:46:49Z

This pull request introduces 13 alerts when merging f5f3cf8 into fe37d3f - view on LGTM.com

new alerts:

4 for Unused import
4 for Modification of parameter with default
2 for Superclass attribute shadows subclass method
1 for Unnecessary pass
1 for Unused local variable
1 for Except block handles 'BaseException'

lgtm-com · 2022-02-11T03:22:23Z

This pull request introduces 6 alerts when merging b665b10 into 64eb620 - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

…_init__.py Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-02-11T04:02:00Z

This pull request introduces 6 alerts when merging 67ab0d2 into 64eb620 - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-02-11T04:41:20Z

This pull request introduces 6 alerts when merging e4418ce into 64eb620 - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-02-11T05:23:17Z

This pull request introduces 6 alerts when merging 012004d into 64eb620 - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-02-11T06:22:45Z

This pull request introduces 6 alerts when merging 8241502 into 64eb620 - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-02-11T06:37:48Z

This pull request introduces 6 alerts when merging 1a1bc0d into 64eb620 - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-02-11T17:33:26Z

This pull request introduces 6 alerts when merging f7a7e9c into 9aef14f - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

yzhang123 · 2022-02-11T00:11:03Z

nemo/collections/nlp/data/dialogue_state_tracking_generative/sgd/assistant_data_processor.py

+        self.data_dir = data_dir
+        self._tokenizer = tokenizer
+
+    def open_file(self, filename):


could you add doc strings for class functions?

lgtm-com · 2022-02-11T17:52:42Z

This pull request introduces 6 alerts when merging c63b2a9 into 9aef14f - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

Signed-off-by: Zhilin Wang <[email protected]>

…/NVIDIA/NeMo into main

lgtm-com · 2022-02-11T18:12:51Z

This pull request introduces 6 alerts when merging b400571 into 9aef14f - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

Signed-off-by: Zhilin Wang <[email protected]>

lgtm-com · 2022-02-11T18:30:44Z

This pull request introduces 6 alerts when merging bd46660 into 9aef14f - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

lgtm-com · 2022-02-11T19:46:20Z

This pull request introduces 6 alerts when merging d100676 into 298b686 - view on LGTM.com

new alerts:

4 for Modification of parameter with default
1 for Unused local variable
1 for Unused import

…ty (#3526) * refactor dialogue state tracking for modelling/dataset interoperability Signed-off-by: Zhilin Wang <[email protected]> * fix style changes Signed-off-by: Zhilin Wang <[email protected]> * fix typo Signed-off-by: Zhilin Wang <[email protected]> * fix style raised by lgtm Signed-off-by: Zhilin Wang <[email protected]> * fix style formatting Signed-off-by: Zhilin Wang <[email protected]> * update template to include description of intent Signed-off-by: Zhilin Wang <[email protected]> * update Jenkinsfile Signed-off-by: Zhilin Wang <[email protected]> * changes based on requests in review Signed-off-by: Zhilin Wang <[email protected]> * add compatibility with assistant dataset Signed-off-by: Zhilin Wang <[email protected]> * update Jenkins Signed-off-by: Zhilin Wang <[email protected]> * remove dialogue_state_tracking Signed-off-by: Zhilin Wang <[email protected]> * update huggingface utils for dialogue Signed-off-by: Zhilin Wang <[email protected]> * rename dialogue_state_tracking_hybrid to dialogue_state_tracking_sgdqa Signed-off-by: Zhilin Wang <[email protected]> * style fix Signed-off-by: Zhilin Wang <[email protected]> * fix style Signed-off-by: Zhilin Wang <[email protected]> * style fix nemo/collections/nlp/models/dialogue_state_tracking_sgdqa/__init__.py Signed-off-by: Zhilin Wang <[email protected]> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <[email protected]> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <[email protected]> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <[email protected]> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <[email protected]> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <[email protected]> * fix typo Signed-off-by: Zhilin Wang <[email protected]> * add docstrings for assistant data processsor Signed-off-by: Zhilin Wang <[email protected]> Co-authored-by: Zhilin Wang <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Co-authored-by: Yang Zhang <[email protected]>

refactor dialogue state tracking for modelling/dataset interoperability

b287422

Signed-off-by: Zhilin Wang <[email protected]>

Zhilin123 requested review from yzhang123, okuchaiev and cparisien January 26, 2022 23:11

okuchaiev reviewed Jan 27, 2022

View reviewed changes

nemo/collections/nlp/modules/common/gpt_module.py Outdated Show resolved Hide resolved

Zhilin123 and others added 3 commits January 27, 2022 16:00

fix style changes

39dbec3

Signed-off-by: Zhilin Wang <[email protected]>

Merge branch 'main' into dialogue_state_tracking_refactor

4bd1cc1

Merge branch 'dialogue_state_tracking_refactor' of https://github.com…

7c12ab6

…/NVIDIA/NeMo into main

Zhilin123 added 2 commits January 31, 2022 10:32

fix typo

23f33f9

Signed-off-by: Zhilin Wang <[email protected]>

fix style raised by lgtm

255294d

Signed-off-by: Zhilin Wang <[email protected]>

fix style formatting

5f6dbd9

Signed-off-by: Zhilin Wang <[email protected]>

Merge branch 'main' into dialogue_state_tracking_refactor

deeeaec

Zhilin123 requested a review from okuchaiev January 31, 2022 23:23

cparisien previously approved these changes Feb 1, 2022

View reviewed changes

Zhilin123 and others added 2 commits February 1, 2022 14:24

update template to include description of intent

4e08423

Signed-off-by: Zhilin Wang <[email protected]>

Merge branch 'main' into dialogue_state_tracking_refactor

7d96cf2

Merge branch 'dialogue_state_tracking_refactor' of https://github.com…

0b1bc6c

…/NVIDIA/NeMo into main

Zhilin123 dismissed cparisien’s stale review via 0b1bc6c February 2, 2022 00:56

Merge branch 'main' into dialogue_state_tracking_refactor

f5f3cf8

style fix nemo/collections/nlp/models/dialogue_state_tracking_sgdqa/_…

67ab0d2

…_init__.py Signed-off-by: Zhilin Wang <[email protected]>

update Jenkinsfile for SGDGEN

e4418ce

Signed-off-by: Zhilin Wang <[email protected]>

update Jenkinsfile for SGDGEN

012004d

Signed-off-by: Zhilin Wang <[email protected]>

update Jenkinsfile for SGDGEN

8241502

Signed-off-by: Zhilin Wang <[email protected]>

update Jenkinsfile for SGDGEN

1a1bc0d

Signed-off-by: Zhilin Wang <[email protected]>

update Jenkinsfile for SGDGEN

f7a7e9c

Signed-off-by: Zhilin Wang <[email protected]>

yzhang123 previously approved these changes Feb 11, 2022

View reviewed changes

Merge branch 'main' into dialogue_state_tracking_refactor

c63b2a9

Zhilin123 added 2 commits February 11, 2022 10:01

fix typo

85a342d

Signed-off-by: Zhilin Wang <[email protected]>

Merge branch 'dialogue_state_tracking_refactor' of https://github.com…

b400571

…/NVIDIA/NeMo into main

Zhilin123 dismissed yzhang123’s stale review via b400571 February 11, 2022 18:02

add docstrings for assistant data processsor

bd46660

Signed-off-by: Zhilin Wang <[email protected]>

Merge branch 'main' into dialogue_state_tracking_refactor

d100676

Zhilin123 requested a review from yzhang123 February 11, 2022 19:35

yzhang123 approved these changes Feb 11, 2022

View reviewed changes

yzhang123 merged commit 058fa38 into main Feb 11, 2022

yzhang123 deleted the dialogue_state_tracking_refactor branch February 11, 2022 20:46

Zhilin123 restored the dialogue_state_tracking_refactor branch February 14, 2022 22:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor dialogue state tracking for modelling/dataset interoperability #3526

refactor dialogue state tracking for modelling/dataset interoperability #3526

Zhilin123 commented Jan 26, 2022 •

edited

Loading

Zhilin123 commented Jan 26, 2022 •

edited

Loading

lgtm-com bot commented Jan 26, 2022

lgtm-com bot commented Jan 28, 2022

lgtm-com bot commented Jan 31, 2022

lgtm-com bot commented Jan 31, 2022

lgtm-com bot commented Jan 31, 2022

okuchaiev commented Feb 1, 2022

okuchaiev commented Feb 1, 2022

cparisien left a comment

lgtm-com bot commented Feb 1, 2022

lgtm-com bot commented Feb 2, 2022

lgtm-com bot commented Feb 2, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

yzhang123 Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

refactor dialogue state tracking for modelling/dataset interoperability #3526

refactor dialogue state tracking for modelling/dataset interoperability #3526

Conversation

Zhilin123 commented Jan 26, 2022 • edited Loading

Zhilin123 commented Jan 26, 2022 • edited Loading

lgtm-com bot commented Jan 26, 2022

lgtm-com bot commented Jan 28, 2022

lgtm-com bot commented Jan 31, 2022

lgtm-com bot commented Jan 31, 2022

lgtm-com bot commented Jan 31, 2022

okuchaiev commented Feb 1, 2022

okuchaiev commented Feb 1, 2022

cparisien left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Feb 1, 2022

lgtm-com bot commented Feb 2, 2022

lgtm-com bot commented Feb 2, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

yzhang123 Feb 11, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

lgtm-com bot commented Feb 11, 2022

Zhilin123 commented Jan 26, 2022 •

edited

Loading

Zhilin123 commented Jan 26, 2022 •

edited

Loading