Pipeline for Text Generation: GenerationPipeline

[enzoampil]

This PR implements a text generation pipeline, GenerationPipeline, which works on any ModelWithLMHead head, and resolves issue #3728

This pipeline predicts the words that will follow a specified text prompt for autoregressive language models. I've registered it to the pipeline function using gpt2 as the default model_type.

The implementation is based on the approach taken in run_generation.py, which means the forward pass uses the PreTrainedModel.generate() method in modeling_utils.py, as recommended to me by @julien-c and @patrickvonplaten .

Sample code:

# Pip install
# If you're using Google Colab, make sure to reset runtime after installing
!pip install -e git+git://github.com/enzoampil/transformers.git@generation_pipeline#egg=transformers

# Pipeline uses `gpt2` by default
from transformers import pipeline
gpt = pipeline('generation', num_return_sequences=1, length=40)
gpt("Natural language processing is amazing!")

# ["Natural language processing is amazing! Just take a look at these some of the features. Go off and read up on them all…\n\nSay hello to the world of BitLocker with ES2016. It's a game."]

Google Colab tutorial here for running GenerationPipeline for the following LM models:

1. OpenAI GPT
2. OpenAI GPT-2
3. Transformer-XL
4. XML
5. XLNet
6. T5
7. CTRL (colab RAM is too small to read this model)

For context, I also plan to use the above GenerationPipeline for my Humor Generation Bot (issue).

I'm very keen to get feedback for the above, so please let me know if I should change anything, or perform additional steps to bring its quality to an acceptable level.

[patrickvonplaten]

@enzoampil - Sorry for fiddling in your code so much :D
It's actually not as easy as I thought to have the final output correct for XLNet and Transfo-XL. My commits suggestions now should work. You should run make style once they are integrated :-)

[patrickvonplaten]

Maybe we should also add an optional padding argument to the __call__ function that overwrites self.PADDING for XLNet and Transfo-XL @LysandreJik. But we can do this in a separate PR @enzoampil - let's try to merge this one first.

[patrickvonplaten]

Sorry, I forgot to add the max_length as generation task specific params to the XLNet and TransfoXL configs. I will do this now.

Ok added it to the config of Transfo-XL and XLNet

@LysandreJik @thomwolf, we also might want to discuss the default generation params for each model. I think it might e.g. be better to set do_sample=True for all models that can generate.

[LysandreJik]

I don't have any strong opinions on whether we should sample or not; However, I think whatever the choice we should make sure that it is explicit in the pipeline documentation that we may control it from the pipeline directly.

Maybe a link linking to the generate method would do the trick, alongside a small explanation that all kwargs will be passed to this underlying method.

[enzoampil]

@patrickvonplaten Ran make_style and just fixed a minor bug from the generation line I think being accidentally taken out from one of your prior commits. The pipeline seems to work fine now 😄

Also, not sure if this is specific to this PR, but there are tests that are suddenly returning an error for the lines that contain self._create_and_check_torchscript(config, inputs_dict).

Sample error:

_____________ AlbertModelTest.test_torchscript_output_hidden_state _____________
[gw7] linux -- Python 3.7.7 /usr/local/bin/python

self = <tests.test_modeling_albert.AlbertModelTest testMethod=test_torchscript_output_hidden_state>

    def test_torchscript_output_hidden_state(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        config.output_hidden_states = True
>       self._create_and_check_torchscript(config, inputs_dict)

tests/test_modeling_common.py:197: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_modeling_common.py:206: in _create_and_check_torchscript
    model = model_class(config=configs_no_init)
/usr/local/lib/python3.7/site-packages/transformers/modeling_albert.py:455: in __init__
    self.init_weights()
/usr/local/lib/python3.7/site-packages/transformers/modeling_utils.py:392: in init_weights
    self.apply(self._init_weights)
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:289: in apply
    module.apply(fn)
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:289: in apply
    module.apply(fn)
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:290: in apply
    fn(self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = AlbertModel(
  (embeddings): AlbertEmbeddings(
    (word_embeddings): Embedding(99, 128, padding_idx=0)
    (position_...     )
      )
    )
  )
  (pooler): Linear(in_features=36, out_features=36, bias=True)
  (pooler_activation): Tanh()
)
module = Embedding(99, 128, padding_idx=0)

    def _init_weights(self, module):
        """ Initialize the weights.
        """
        if isinstance(module, (nn.Linear, nn.Embedding)):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
>           module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
E           RuntimeError: normal_ expects std > 0.0, but found std=0

/usr/local/lib/python3.7/site-packages/transformers/modeling_albert.py:377: RuntimeError
________________________ BertModelTest.test_headmasking ________________________
[gw1] linux -- Python 3.7.7 /usr/local/bin/python

self = <tests.test_modeling_bert.BertModelTest testMethod=test_headmasking>

    def test_headmasking(self):
        if not self.test_head_masking:
            return
    
        global_rng.seed(42)
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
        global_rng.seed()
    
        config.output_attentions = True
        config.output_hidden_states = True
        configs_no_init = _config_zero_init(config)  # To be sure we have no Nan
        for model_class in self.all_model_classes:
>           model = model_class(config=configs_no_init)

tests/test_modeling_common.py:260: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.7/site-packages/transformers/modeling_bert.py:619: in __init__
    self.init_weights()
/usr/local/lib/python3.7/site-packages/transformers/modeling_utils.py:392: in init_weights
    self.apply(self._init_weights)
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:289: in apply
    module.apply(fn)
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:289: in apply
    module.apply(fn)
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:290: in apply
    fn(self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = BertModel(
  (embeddings): BertEmbeddings(
    (word_embeddings): Embedding(99, 32, padding_idx=0)
    (position_embed...
  (pooler): BertPooler(
    (dense): Linear(in_features=32, out_features=32, bias=True)
    (activation): Tanh()
  )
)
module = Embedding(99, 32, padding_idx=0)

    def _init_weights(self, module):
        """ Initialize the weights """
        if isinstance(module, (nn.Linear, nn.Embedding)):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
>           module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
E           RuntimeError: normal_ expects std > 0.0, but found std=0

/usr/local/lib/python3.7/site-packages/transformers/modeling_bert.py:525: RuntimeError

[patrickvonplaten]

Those test are probably falling because the new Pytorch version was released. Can you just tense your branch in master?:

$ git fetch upstream
$ git rebase upstream/master

(Assuming that you added the master branch as a remote branch "upstream").

The test should then pass :-)

[enzoampil]

@patrickvonplaten Apologies, I'm having issues with the rebase suggested above.

I initially tried it but ended up showing up as a co-committer with the rebased commits, which explains why I performed a force-push above to revert the rebase. It might be related to an issue I'm having where I'm forced to do a rebase --skip with each of the conflicts (same situation as here).

May I please ask for some assistance / advice with this?

[julien-c]

Clean implementation, congrats @enzoampil and thanks @patrickvonplaten for the great review

[enzoampil]

Once again, thanks so much! Looking forward to contributing more in the future 😄@patrickvonplaten @julien-c