Attention weight logging

[JEM-Mosig]

Proposed changes:

• Closes https://github.com/RasaHQ/research/issues/73

Status (please check what you already did):

• added some tests for the functionality
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)
• List this as experimental feature here

[tmbo]

@JEM-Mosig what's up with this PR?

[JEM-Mosig]

@JEM-Mosig what's up with this PR?

The corresponding project is on hold due to deadline for dataset collection. Should continue mid June.

[tmbo]

how is the implementation going, is this still relevant @JEM-Mosig ?

[JEM-Mosig]

how is the implementation going, is this still relevant @JEM-Mosig ?

I'm still on this. It's a side project and might take a bit more time. Not critical for 2.0 or anything. I only created a draft PR so its easier for others to see the code changes when we discuss.

[koaning]

@JEM-Mosig I just had a look and it's making me wonder what the easiest way might to go about visualising this. Your PR opens up the attention weights but there's also some other vectors that I'd love to get a hold on too.

I'm wondering what is practical here. I'd love to make DIET more "peekable" but doing so might add unwanted complexity.

[Ghostvv]

@JEM-Mosig why do we need special ..._with_diagnostic methods? I don't see where it is called in processor. Can we just reuse normal predict and process methods?

[JEM-Mosig]

@JEM-Mosig why do we need special ..._with_diagnostic methods? I don't see where it is called in processor. Can we just reuse normal predict and process methods?

If we output the attention weights (and maybe other data) with the predict / process functions, then messages get a lot bigger. Though we could add a parameter that decides to add or not to add the diagnostic data. I think I'll change it to be parameters @koaning .
Right now, the _with_diagnostic functions are only called by rasalit.

[Ghostvv]

I see. @wochinge what do you think would be the best approach here?