Attention layer output

[krayush07]

The method task_specific_attention applies attention to the projected vectors instead of the hidden vectors (output from RNN cell).

Has it been applied purposefully or has the information on attention according to the paper been missed out where final sentence vector is weighted summation of hidden states and NOT inner projected vector?

[diegow88]

@krayush07 you are right. Have you tested the implementation by changing the attention to the hidden vectors?

[heisenbugfix]

@krayush07 - In the paper this is how the projection is done:
Lets say your hidden state output is h_it. Then you first calculate u_it as:
u_it = tanh(W*h_it + b_w)
Then attention weights are calculated using u_it. So u_it here is the projected vector

[krayush07]

@heisenbugfix I agree to what you mentioned in the previous comment. However, final attention is applied on hidden states and NOT projected vector.

As per my understanding, here are the steps apply attention:

1. Collect 'hidden_states'
2. Apply projection to get projected vector.
3. Use projected vector and attention vector to find attention weights.
4. Use attention weights and hidden_states to apply attention.

I find a mismatch in 4th step in your code. Please correct me if I am wrong.

[heisenbugfix]

@krayush07 Ah I get it. Thanks for clarifying.
P.S It is not my code :D although its an awesome code.

[momih]

@krayush07 I think it's more of a personal choice where to apply attention weights. In the paper, the authors project the hidden state to the same dimension and then compute attention and apply it to the hidden state. However, in this implementation he projects the hidden state to a lower dimension to compute attention.
So I'm guessing he applies attention to the projected vector instead because he wants a lower dimension for the encoded sentence vector.