Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check if training entities in DIET is still working on e2e #7289

Closed
Ghostvv opened this issue Nov 16, 2020 · 6 comments
Closed

check if training entities in DIET is still working on e2e #7289

Ghostvv opened this issue Nov 16, 2020 · 6 comments
Assignees
Labels
area:rasa-oss/ml 👁 All issues related to machine learning type:enhancement ✨ Additions of new features or changes to existing ones, should be doable in a single PR
Milestone

Comments

@Ghostvv
Copy link
Contributor

Ghostvv commented Nov 16, 2020

On e2e, we updated tf methods for TED, since these methods are reused in DIET, DIET performance on e2e branch should be benchmarked against DIET in master

@Ghostvv Ghostvv added type:enhancement ✨ Additions of new features or changes to existing ones, should be doable in a single PR area:rasa-oss 🎡 Anything related to the open source Rasa framework labels Nov 16, 2020
@Ghostvv Ghostvv added this to the e2e milestone Nov 16, 2020
@tmbo tmbo added area:rasa-oss/ml 👁 All issues related to machine learning and removed area:rasa-oss 🎡 Anything related to the open source Rasa framework labels Nov 17, 2020
@tttthomasssss
Copy link
Contributor

@Ghostvv do you have any datasets in mind? Otherwise I would have gone for carbon-bot as a start, and then potentially some of the ones we've used in the DIET paper.

@Ghostvv
Copy link
Contributor Author

Ghostvv commented Nov 25, 2020

@tttthomasssss I'd start by running all our datasets through GitHub as we do it for PRs

tttthomasssss added a commit that referenced this issue Nov 25, 2020
@tttthomasssss
Copy link
Contributor

@tttthomasssss
Copy link
Contributor

@Ghostvv results look pretty similar, the ones from e2e seem to be a tiny bit lower than from master (on carbon-bot and HERMIT), though I'd regard them within range still. The stuff from e2e also seems to be a bit slower (e.g. up to 2mins for carbon-bot and up to 5mins for HERMIT). But this might also due to github.

@Ghostvv
Copy link
Contributor Author

Ghostvv commented Nov 27, 2020

Great! then I think we can close the issue, @tabergma do you have any other tests in mind?

@tabergma
Copy link
Contributor

Great, no, I think that should be sufficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:rasa-oss/ml 👁 All issues related to machine learning type:enhancement ✨ Additions of new features or changes to existing ones, should be doable in a single PR
Projects
None yet
Development

No branches or pull requests

4 participants