added transformers_config for passing arguments to the transformer

[KennethEnevoldsen]

Added transformers_config to allow the user to pass arguments to the transformers forward pass. Most notably the output_attentions.

for convenience, I used this example to test the code:

import spacy

nlp = spacy.blank("en")

# Construction via add_pipe with custom config
config = {
    "model": {
        "@architectures": "spacy-transformers.TransformerModel.v1",
        "name": "bert-base-uncased",
        "transformers_config": {"output_attentions": True},
    }
}
transformer = nlp.add_pipe(
    "transformer", config=config)
transformer.model.initialize()


doc = nlp("This is a sentence.")

which gives you:

len(doc._.trf_data.attention) # 12
doc._.trf_data.attention[-1].shape  # (1, 12, 7, 7) last layer of attention 
len(doc._.trf_data.tensors) # 2 
doc._.trf_data.tensors[0].shape # (1, 7, 768) <-- wordpiece embedding
doc._.trf_data.tensors[1].shape  # (1, 768) <-- assuming this is the pooled embedding?

Sidenote: it took me quite a while to find the default config str. It might be ideal to make this into a standalone file and load it in?

[honnibal]

@KennethEnevoldsen Thanks for this, and sorry for the delay reviewing it. I think it'll be a really nice feature, it's just a question of getting these details about the backwards compat right.

[KennethEnevoldsen]

Thanks, @honnibal, and completely understandable. It is potentially a feature that could be integrated nicely with displacy for displaying the attention given to each word. Not entirely sure though seems like the judge is still out on how interpretability these attention weights are.

[svlandeg]

It looks like a binary file ".DS_Store" got committed by accident...

[svlandeg]

@KennethEnevoldsen : Apologies again for the late follow-up! I just did one final round of review and think we should be able to wrap this up soon :-)

[KennethEnevoldsen]

No problem will follow up on this during the weekend

[svlandeg]

Thanks again for your contribution and patience! I think this looks good - just made an internal note about the documentation & moving v1 to legacy, we'll take care of that in a separate PR.