diff --git a/Makefile b/Makefile
index 21d11434cc47..36e9d0aea77b 100644
--- a/Makefile
+++ b/Makefile
@@ -1,5 +1,7 @@
.PHONY: deps_table_update modified_only_fixup extra_quality_checks quality style fixup fix-copies test test-examples docs
+# make sure to test the local checkout in scripts and not the pre-installed one (don't use quotes!)
+export PYTHONPATH = src
check_dirs := examples tests src utils
diff --git a/README.md b/README.md
index 1e7ced945aa8..6b3208d6c648 100644
--- a/README.md
+++ b/README.md
@@ -38,14 +38,14 @@ limitations under the License.
-
State-of-the-art Natural Language Processing for PyTorch and TensorFlow 2.0
+
State-of-the-art Natural Language Processing for Jax, PyTorch and TensorFlow
🤗 Transformers provides thousands of pretrained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, text generation, etc in 100+ languages. Its aim is to make cutting-edge NLP easier to use for everyone.
🤗 Transformers provides APIs to quickly download and use those pretrained models on a given text, fine-tune them on your own datasets then share them with the community on our [model hub](https://huggingface.co/models). At the same time, each python module defining an architecture can be used as a standalone and modified to enable quick research experiments.
-🤗 Transformers is backed by the two most popular deep learning libraries, [PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/), with a seamless integration between them, allowing you to train your models with one then load it for inference with the other.
+🤗 Transformers is backed by the three most popular deep learning libraries, [Jax](https://jax.readthedocs.io/en/latest/), [PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/), with a seamless integration between them, allowing you to train your models with one then load it for inference with the other.
## Online demos
@@ -152,16 +152,16 @@ The model itself is a regular [Pytorch `nn.Module`](https://pytorch.org/docs/sta
### With pip
-This repository is tested on Python 3.6+, PyTorch 1.0.0+ (PyTorch 1.3.1+ for [examples](https://github.com/huggingface/transformers/tree/master/examples)) and TensorFlow 2.0.
+This repository is tested on Python 3.6+, Flax 0.3.2+, PyTorch 1.3.1+ and TensorFlow 2.3+.
You should install 🤗 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).
First, create a virtual environment with the version of Python you're going to use and activate it.
-Then, you will need to install at least one of TensorFlow 2.0, PyTorch or Flax.
-Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available), [PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific install command for your platform and/or [Flax installation page](https://github.com/google/flax#quick-install).
+Then, you will need to install at least one of Flax, PyTorch or TensorFlow.
+Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/), [PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific install command for your platform and/or [Flax installation page](https://github.com/google/flax#quick-install).
-When TensorFlow 2.0 and/or PyTorch has been installed, 🤗 Transformers can be installed using pip as follows:
+When one of those backends has been installed, 🤗 Transformers can be installed using pip as follows:
```bash
pip install transformers
@@ -179,7 +179,7 @@ Since Transformers version v4.0.0, we now have a conda channel: `huggingface`.
conda install -c huggingface transformers
```
-Follow the installation pages of TensorFlow, PyTorch or Flax to see how to install them with conda.
+Follow the installation pages of Flax, PyTorch or TensorFlow to see how to install them with conda.
## Models architectures
@@ -247,7 +247,7 @@ Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.
1. **[XLSR-Wav2Vec2](https://huggingface.co/transformers/model_doc/xlsr_wav2vec2.html)** (from Facebook AI) released with the paper [Unsupervised Cross-Lingual Representation Learning For Speech Recognition](https://arxiv.org/abs/2006.13979) by Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael Auli.
1. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
-To check if each model has an implementation in PyTorch/TensorFlow/Flax or has an associated tokenizer backed by the 🤗 Tokenizers library, refer to [this table](https://huggingface.co/transformers/index.html#bigtable)
+To check if each model has an implementation in Flax, PyTorch or TensorFlow, or has an associated tokenizer backed by the 🤗 Tokenizers library, refer to [this table](https://huggingface.co/transformers/index.html#bigtable)
These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations. You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html).
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 25a2a380431e..8fc8700a0b5b 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -1,12 +1,12 @@
Transformers
=======================================================================================================================
-State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.
+State-of-the-art Natural Language Processing for Jax, Pytorch and TensorFlow
🤗 Transformers (formerly known as `pytorch-transformers` and `pytorch-pretrained-bert`) provides general-purpose
architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet...) for Natural Language Understanding (NLU) and Natural
-Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between
-TensorFlow 2.0 and PyTorch.
+Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between Jax,
+PyTorch and TensorFlow.
This is the documentation of our repository `transformers `_.
@@ -43,11 +43,11 @@ Lower compute costs, smaller carbon footprint:
Choose the right framework for every part of a model's lifetime:
- Train state-of-the-art models in 3 lines of code
-- Deep interoperability between TensorFlow 2.0 and PyTorch models
-- Move a single model between TF2.0/PyTorch frameworks at will
+- Deep interoperability between Jax, Pytorch and TensorFlow models
+- Move a single model between Jax/PyTorch/TensorFlow frameworks at will
- Seamlessly pick the right framework for training, evaluation, production
-Experimental support for Flax with a few models right now, expected to grow in the coming months.
+The support for Jax is still experimental (with a few models right now), expect to see it grow in the coming months!
`All the model checkpoints `__ are seamlessly integrated from the huggingface.co `model
hub `__ where they are uploaded directly by `users `__ and
@@ -74,8 +74,8 @@ The documentation is organized in five parts:
- **MODELS** for the classes and functions related to each model implemented in the library.
- **INTERNAL HELPERS** for the classes and functions we use internally.
-The library currently contains PyTorch, Tensorflow and Flax implementations, pretrained model weights, usage scripts
-and conversion utilities for the following models:
+The library currently contains Jax, PyTorch and Tensorflow implementations, pretrained model weights, usage scripts and
+conversion utilities for the following models:
..
This list is updated automatically from the README with `make fix-copies`. Do not update manually!
@@ -251,8 +251,8 @@ and conversion utilities for the following models:
.. _bigtable:
The table below represents the current support in the library for each of those models, whether they have a Python
-tokenizer (called "slow"). A "fast" tokenizer backed by the 🤗 Tokenizers library, whether they have support in PyTorch,
-TensorFlow and/or Flax.
+tokenizer (called "slow"). A "fast" tokenizer backed by the 🤗 Tokenizers library, whether they have support in Jax (via
+Flax), PyTorch, and/or TensorFlow.
..
This table is updated automatically from the auto modules with `make fix-copies`. Do not update manually!
diff --git a/docs/source/installation.md b/docs/source/installation.md
index eecd48de338a..1b7d8d5d5911 100644
--- a/docs/source/installation.md
+++ b/docs/source/installation.md
@@ -149,12 +149,6 @@ So if you don't have any specific environment variable set, the cache directory
(``PYTORCH_TRANSFORMERS_CACHE`` or ``PYTORCH_PRETRAINED_BERT_CACHE``), those will be used if there is no shell
environment variable for ``TRANSFORMERS_CACHE``.
-### Note on model downloads (Continuous Integration or large-scale deployments)
-
-If you expect to be downloading large volumes of models (more than 10,000) from huggingface.co (for instance through
-your CI setup, or a large-scale production deployment), please cache the model files on your end. It will be way
-faster, and cheaper. Feel free to contact us privately, we'd love to help with this.
-
### Offline mode
It's possible to run 🤗 Transformers in a firewalled or a no-network environment.
diff --git a/docs/source/main_classes/output.rst b/docs/source/main_classes/output.rst
index 7b7c05568a45..a627571f2413 100644
--- a/docs/source/main_classes/output.rst
+++ b/docs/source/main_classes/output.rst
@@ -13,8 +13,8 @@
Model outputs
-----------------------------------------------------------------------------------------------------------------------
-PyTorch models have outputs that are instances of subclasses of :class:`~transformers.file_utils.ModelOutput`. Those
-are data structures containing all the information returned by the model, but that can also be used as tuples or
+All models have outputs that are instances of subclasses of :class:`~transformers.file_utils.ModelOutput`. Those are
+data structures containing all the information returned by the model, but that can also be used as tuples or
dictionaries.
Let's see of this looks on an example:
diff --git a/docs/source/main_classes/trainer.rst b/docs/source/main_classes/trainer.rst
index 106ef3c80ef8..cdc796c017de 100644
--- a/docs/source/main_classes/trainer.rst
+++ b/docs/source/main_classes/trainer.rst
@@ -400,18 +400,18 @@ DeepSpeed
`DeepSpeed `__ implements everything described in the `ZeRO paper
`__. Currently it provides full support for:
-1. Optimizer State Partitioning (ZeRO stage 1)
-2. Gradient Partitioning (ZeRO stage 2)
-3. Param Partitioning (ZeRO stage 3)
+1. Optimizer state partitioning (ZeRO stage 1)
+2. Gradient partitioning (ZeRO stage 2)
+3. Parameter partitioning (ZeRO stage 3)
4. Custom mixed precision training handling
-5. A range of fast CUDA-extension-based Optimizers
-6. ZeRO-Offload
+5. A range of fast CUDA-extension-based optimizers
+6. ZeRO-Offload to CPU and NVMe
ZeRO-Offload has its own dedicated paper: `ZeRO-Offload: Democratizing Billion-Scale Model Training
-`__.
+`__. And NVMe-support is described in the paper `ZeRO-Infinity: Breaking the GPU
+Memory Wall for Extreme Scale Deep Learning `__.
-DeepSpeed ZeRO-2 is currently used only for training, as all the currently available features are of no use to
-inference.
+DeepSpeed ZeRO-2 is primarily used only for training, as its features are of no use to inference.
DeepSpeed ZeRO-3 can be used for inference as well, since it allows huge models to be loaded on multiple GPUs, which
won't be possible on a single GPU.
@@ -541,7 +541,7 @@ Here is an example of running ``run_translation.py`` under DeepSpeed deploying a
.. code-block:: bash
deepspeed examples/pytorch/translation/run_translation.py \
- --deepspeed tests/deepspeed/ds_config.json \
+ --deepspeed tests/deepspeed/ds_config_zero3.json \
--model_name_or_path t5-small --per_device_train_batch_size 1 \
--output_dir output_dir --overwrite_output_dir --fp16 \
--do_train --max_train_samples 500 --num_train_epochs 1 \
@@ -566,17 +566,17 @@ To deploy DeepSpeed with one GPU adjust the :class:`~transformers.Trainer` comma
.. code-block:: bash
deepspeed --num_gpus=1 examples/pytorch/translation/run_translation.py \
- --deepspeed tests/deepspeed/ds_config.json \
+ --deepspeed tests/deepspeed/ds_config_zero2.json \
--model_name_or_path t5-small --per_device_train_batch_size 1 \
--output_dir output_dir --overwrite_output_dir --fp16 \
--do_train --max_train_samples 500 --num_train_epochs 1 \
--dataset_name wmt16 --dataset_config "ro-en" \
--source_lang en --target_lang ro
-This is almost the same as with multiple-GPUs, but here we tell DeepSpeed explicitly to use just one GPU. By default,
-DeepSpeed deploys all GPUs it can see. If you have only 1 GPU to start with, then you don't need this argument. The
-following `documentation `__ discusses the
-launcher options.
+This is almost the same as with multiple-GPUs, but here we tell DeepSpeed explicitly to use just one GPU via
+``--num_gpus=1``. By default, DeepSpeed deploys all GPUs it can see on the given node. If you have only 1 GPU to start
+with, then you don't need this argument. The following `documentation
+`__ discusses the launcher options.
Why would you want to use DeepSpeed with just one GPU?
@@ -601,7 +601,7 @@ with DeepSpeed is to have at least the following configuration in the configurat
"overlap_comm": true,
"contiguous_gradients": true,
"cpu_offload": true
- },
+ }
}
which enables ``cpu_offload`` and some other important features. You may experiment with the buffer sizes, you will
@@ -610,6 +610,11 @@ find more details in the discussion below.
For a practical usage example of this type of deployment, please, see this `post
`__.
+You may also try the ZeRO-3 with CPU and NVMe offload as explained further in this document.
+
+
+
Notes:
- if you need to run on a specific GPU, which is different from GPU 0, you can't use ``CUDA_VISIBLE_DEVICES`` to limit
@@ -643,7 +648,7 @@ If you're using only 1 GPU, here is how you'd have to adjust your training code
os.environ['WORLD_SIZE'] = "1"
# Now proceed as normal, plus pass the deepspeed config file
- training_args = TrainingArguments(..., deepspeed="ds_config.json")
+ training_args = TrainingArguments(..., deepspeed="ds_config_zero3.json")
trainer = Trainer(...)
trainer.train()
@@ -659,47 +664,62 @@ cell with:
.. code-block:: python
%%bash
- cat <<'EOT' > ds_config.json
+ cat <<'EOT' > ds_config_zero3.json
{
"fp16": {
- "enabled": true,
+ "enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
+ "initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
- "zero_optimization": {
- "stage": 2,
- "allgather_partitions": true,
- "allgather_bucket_size": 2e8,
- "overlap_comm": true,
- "reduce_scatter": true,
- "reduce_bucket_size": 2e8,
- "contiguous_gradients": true,
- "cpu_offload": true
- },
-
"optimizer": {
"type": "AdamW",
"params": {
- "lr": 3e-5,
- "betas": [0.8, 0.999],
- "eps": 1e-8,
- "weight_decay": 3e-7
+ "lr": "auto",
+ "betas": "auto",
+ "eps": "auto",
+ "weight_decay": "auto"
}
},
"scheduler": {
"type": "WarmupLR",
"params": {
- "warmup_min_lr": 0,
- "warmup_max_lr": 3e-5,
- "warmup_num_steps": 500
+ "warmup_min_lr": "auto",
+ "warmup_max_lr": "auto",
+ "warmup_num_steps": "auto"
}
},
+ "zero_optimization": {
+ "stage": 3,
+ "offload_optimizer": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "offload_param": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "overlap_comm": true,
+ "contiguous_gradients": true,
+ "sub_group_size": 1e14,
+ "reduce_bucket_size": "auto",
+ "stage3_prefetch_bucket_size": "auto",
+ "stage3_param_persistence_threshold": "auto",
+ "stage3_max_live_parameters": 1e9,
+ "stage3_max_reuse_distance": 1e9,
+ "stage3_gather_fp16_weights_on_model_save": true
+ },
+
+ "gradient_accumulation_steps": "auto",
+ "gradient_clipping": "auto",
"steps_per_print": 2000,
+ "train_batch_size": "auto",
+ "train_micro_batch_size_per_gpu": "auto",
"wall_clock_breakdown": false
}
EOT
@@ -725,7 +745,7 @@ or with ``%%bash`` magic, where you can write a multi-line code for the shell pr
In such case you don't need any of the code presented at the beginning of this section.
-Note: ``%%bash`` magic is neat, but currently it buffers the output so you won't see the logs until the process
+Note: While ``%%bash`` magic is neat, but currently it buffers the output so you won't see the logs until the process
completes.
@@ -760,48 +780,55 @@ When using DeepSpeed you always need to supply a DeepSpeed configuration file, y
to be configured via the command line. You will find the nuances in the rest of this guide.
To get an idea of what DeepSpeed configuration file looks like, here is one that activates ZeRO stage 2 features,
-enables FP16, uses ``AdamW`` optimizer and ``WarmupLR`` scheduler:
+including optimizer states cpu offload, uses ``AdamW`` optimizer and ``WarmupLR`` scheduler and will enable mixed
+precision training if ``--fp16`` is passed:
.. code-block:: json
{
"fp16": {
- "enabled": true,
+ "enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
+ "initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
- "zero_optimization": {
- "stage": 2,
- "allgather_partitions": true,
- "allgather_bucket_size": 5e8,
- "overlap_comm": true,
- "reduce_scatter": true,
- "reduce_bucket_size": 5e8,
- "contiguous_gradients": true,
- "cpu_offload": true
- },
+ "optimizer": {
+ "type": "AdamW",
+ "params": {
+ "lr": "auto",
+ "betas": "auto",
+ "eps": "auto",
+ "weight_decay": "auto"
+ }
+ },
- "optimizer": {
- "type": "AdamW",
- "params": {
- "lr": 3e-5,
- "betas": [ 0.8, 0.999 ],
- "eps": 1e-8,
- "weight_decay": 3e-7
- }
- },
+ "scheduler": {
+ "type": "WarmupLR",
+ "params": {
+ "warmup_min_lr": "auto",
+ "warmup_max_lr": "auto",
+ "warmup_num_steps": "auto"
+ }
+ },
- "scheduler": {
- "type": "WarmupLR",
- "params": {
- "warmup_min_lr": 0,
- "warmup_max_lr": 3e-5,
- "warmup_num_steps": 500
- }
- }
+ "zero_optimization": {
+ "stage": 2,
+ "allgather_partitions": true,
+ "allgather_bucket_size": 2e8,
+ "overlap_comm": true,
+ "reduce_scatter": true,
+ "reduce_bucket_size": 2e8,
+ "contiguous_gradients": true,
+ "cpu_offload": true
+ },
+
+ "gradient_accumulation_steps": "auto",
+ "gradient_clipping": "auto",
+ "train_batch_size": "auto",
+ "train_micro_batch_size_per_gpu": "auto",
}
When you execute the program, DeepSpeed will log the configuration it received from the :class:`~transformers.Trainer`
@@ -835,35 +862,38 @@ or:
Shared Configuration
=======================================================================================================================
-Some configuration information is required by both the :class:`~transformers.Trainer` and DeepSpeed to function
-correctly, therefore, to prevent conflicting definitions, which could lead to hard to detect errors, we chose to
-configure those via the :class:`~transformers.Trainer` command line arguments.
-
-Therefore, the following DeepSpeed configuration params shouldn't be used with the :class:`~transformers.Trainer`:
-* ``train_batch_size``
-* ``train_micro_batch_size_per_gpu``
-* ``gradient_accumulation_steps``
+.. warning::
-as these will be automatically derived from the run time environment and the following 2 command line arguments:
+ This section is a must-read
-.. code-block:: bash
+Some configuration values are required by both the :class:`~transformers.Trainer` and DeepSpeed to function correctly,
+therefore, to prevent conflicting definitions, which could lead to hard to detect errors, we chose to configure those
+via the :class:`~transformers.Trainer` command line arguments.
- --per_device_train_batch_size 8 --gradient_accumulation_steps 2
+Additionally, some configuration values are derived automatically based on the model's configuration, so instead of
+remembering to manually adjust multiple values, it's the best to let the :class:`~transformers.Trainer` do the majority
+of configuration for you.
-which are always required to be supplied.
+Therefore, in the rest of this guide you will find a special configuration value: ``auto``, which when set will be
+automatically replaced with the correct or most efficient value. Please feel free to choose to ignore this
+recommendation and set the values explicitly, in which case be very careful that your the
+:class:`~transformers.Trainer` arguments and DeepSpeed configurations agree. For example, are you using the same
+learning rate, or batch size, or gradient accumulation settings? if these mismatch the training may fail in very
+difficult to detect ways. You have been warned.
-Of course, you will need to adjust the values in this example to your situation.
+There are multiple other values that are specific to DeepSpeed-only and those you will have to set manually to suit
+your needs.
ZeRO
=======================================================================================================================
-`Zero Redundancy Optimizer (ZeRO) `__ is the work horse of DeepSpeed. It
+`Zero Redundancy Optimizer (ZeRO) `__ is the workhorse of DeepSpeed. It
support 3 different levels (stages) of optimization. The first one is not quite interesting for scalability purposes,
-therefore this document focuses on stages 2 and 3. You will find more indepth information in the DeepSpeed
-documentation.
+therefore this document focuses on stages 2 and 3. Stage 3 is further improved by the latest addition of ZeRO-Infinity.
+You will find more indepth information in the DeepSpeed documentation.
The ``zero_optimization`` section of the configuration file is the most important part (`docs
`__), since that is where you define
@@ -916,36 +946,43 @@ ZeRO-3 Config
The following is an example configuration for ZeRO stage 3:
-
.. code-block:: json
{
"zero_optimization": {
"stage": 3,
- "cpu_offload": true,
- "cpu_offload_params": true,
- "cpu_offload_use_pin_memory" : true,
+ "offload_optimizer": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "offload_param": {
+ "device": "cpu",
+ "pin_memory": true
+ },
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1e14,
- "reduce_bucket_size": 1e6,
- "stage3_prefetch_bucket_size": 0.94e6,
- "stage3_param_persistence_threshold": 1e4,
+ "reduce_bucket_size": "auto",
+ "stage3_prefetch_bucket_size": "auto",
+ "stage3_param_persistence_threshold": "auto",
"stage3_max_live_parameters": 1e9,
"stage3_max_reuse_distance": 1e9,
"stage3_gather_fp16_weights_on_model_save": true
}
}
-Note: if you're migrating from ZeRO-2 configuration that: ``allgather_partitions``, ``allgather_bucket_size`` and
-``reduce_scatter`` configuration parameters are not used in ZeRO-3. If you keep these they will just be ignored.
+If you are getting OOMs, because your model or activations don't fit into the GPU memory and you have unutilized CPU
+memory offloading the optimizer states and parameters to CPU memory with ``"device": "cpu"`` may solve this limitation.
+If you don't want to offload to CPU memory, use ``none`` instead of ``cpu`` for the ``device`` entry. Offloading to
+NVMe is discussed further down.
+
+Pinned memory is enabled with ``pin_memory`` set to ``true``. This feature can improve the throughput at the cost of
+making less memory available to other processes. Pinned memory is set aside to the specific process that requested it
+and its typically accessed much faster than normal CPU memory.
**Performance tuning:**
- ``sub_group_size``: ``1e14``
-- ``reduce_bucket_size``: ``hidden_size*hidden_size``
-- ``stage3_prefetch_bucket_size``: ``0.9 * hidden_size * hidden_size``
-- ``stage3_param_persistence_threshold``: ``10 * hidden_size``
- ``stage3_max_live_parameters``: ``1e9``
- ``stage3_max_reuse_distance``: ``1e9``
@@ -960,37 +997,91 @@ going to be used again in near future (less than ``stage3_max_reuse_distance``)
overhead. This is super helpful when you have activation checkpointing enabled, where we do a forward recompute and
backward passes a a single layer granularity and want to keep the parameter in the forward recompute till the backward
-If you set ``reduce_bucket_size``, ``stage3_prefetch_bucket_size`` and ``stage3_param_persistence_threshold`` as
-recommended above, they will already be fairly small so you won't have to tune those much.
+The following configuration values depend on the model's hidden size:
+
+- ``reduce_bucket_size``: ``hidden_size*hidden_size``
+- ``stage3_prefetch_bucket_size``: ``0.9 * hidden_size * hidden_size``
+- ``stage3_param_persistence_threshold``: ``10 * hidden_size``
+
+therefore set these values to ``auto`` and the :class:`~transformers.Trainer` will automatically assign the recommended
+values. But, of course, feel free to set these explicitly as well.
+
+``stage3_gather_fp16_weights_on_model_save`` enables model fp16 weights consolidation when model gets saved. With large
+models and multiple GPUs this is an expensive operation both in terms of memory and speed. It's currently required if
+you plan to resume the training. Watch out for future updates that will remove this limitation and make things more
+flexible.
+
+If you're migrating from ZeRO-2 configuration note that ``allgather_partitions``, ``allgather_bucket_size`` and
+``reduce_scatter`` configuration parameters are not used in ZeRO-3. If you keep these in the config file they will just
+be ignored. Make sure to remove ``cpu_offload`` though, since it has been deprecated in ZeRO-3.
+
+
+
+
+NVMe Support
+=======================================================================================================================
-Since ``hidden_size`` varies from model to model, the ``Trainer`` will automatically set the needed value for the 3
-config parameters that contain that variable (using ``model.config.hidden_size``). Just set these values to ``0`` as
-shown below and the right configuration will be passed to DeepSpeed:
+ZeRO-Infinity allows for training incredibly large models by extending GPU and CPU memory with NVMe memory. Thanks to
+smart partitioning and tiling algorithms each GPU needs to send and receive very small amounts of data during
+offloading so modern NVMe proved to be fit to allow for an even larger total memory pool available to your training
+process. ZeRO-Infinity requires ZeRO-3 enabled.
+
+The following configuration example enables NVMe to offload both optimizer states and the params:
.. code-block:: json
{
"zero_optimization": {
"stage": 3,
- "cpu_offload": true,
- "cpu_offload_params": true,
- "cpu_offload_use_pin_memory" : true,
+ "offload_optimizer": {
+ "device": "nvme",
+ "nvme_path": "/local_nvme",
+ "pin_memory": true,
+ "buffer_count": 4,
+ "fast_init": false
+ },
+ "offload_param": {
+ "device": "nvme",
+ "nvme_path": "/local_nvme",
+ "pin_memory": true,
+ "buffer_count": 5,
+ "buffer_size": 1e8,
+ "max_in_cpu": 1e9
+ }
+ "aio": {
+ "block_size": 262144,
+ "queue_depth": 32,
+ "thread_count": 1,
+ "single_submit": false,
+ "overlap_events": true
+ }
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1e14,
- "reduce_bucket_size": 0,
- "stage3_prefetch_bucket_size": 0,
- "stage3_param_persistence_threshold": 0,
+ "reduce_bucket_size": "auto",
+ "stage3_prefetch_bucket_size": "auto",
+ "stage3_param_persistence_threshold": "auto",
"stage3_max_live_parameters": 1e9,
"stage3_max_reuse_distance": 1e9,
"stage3_gather_fp16_weights_on_model_save": true
- }
+ },
}
-``stage3_gather_fp16_weights_on_model_save`` enables model fp16 weights consolidation when model gets saved. With large
-models and multiple GPUs this is an expensive operation both in terms of memory and speed. It's currently required if
-you plan to resume the training. Watch out for future updates that will remove this limitation and make things more
-flexible.
+You can choose to offload both optimizer states and params to NVMe, or just one of them or none. For example, if you
+have copious amounts of CPU memory available, by all means offload to CPU memory only as it'd be faster (hint:
+`"device": "cpu"`).
+
+Here is the full documentation for offloading `optimizer states
+`__ and `parameters
+`__.
+
+Make sure that your ``nvme_path`` is actually an NVMe, since it will work with the normal hard drive or SSD, but it'll
+be much much slower. The fast scalable training was designed with modern NVMe transfer speeds in mind (as of this
+writing one can have ~3.5GB/s read, ~3GB/s write peak speeds).
+
+In order to figure out the optimal ``aio`` configuration block you must run a benchmark on your target setup, as
+`explained here `__.
+
ZeRO-2 vs ZeRO-3 Performance
@@ -1016,13 +1107,13 @@ these help you to trade scalability for speed depending on your needs.
ZeRO-2 Example
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-Here is a full ZeRO-2 all-enabled configuration file ``ds_config_zero2.json``:
+Here is a full ZeRO-2 auto-configuration file ``ds_config_zero2.json``:
.. code-block:: json
{
"fp16": {
- "enabled": true,
+ "enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
@@ -1030,6 +1121,25 @@ Here is a full ZeRO-2 all-enabled configuration file ``ds_config_zero2.json``:
"min_loss_scale": 1
},
+ "optimizer": {
+ "type": "AdamW",
+ "params": {
+ "lr": "auto",
+ "betas": "auto",
+ "eps": "auto",
+ "weight_decay": "auto"
+ }
+ },
+
+ "scheduler": {
+ "type": "WarmupLR",
+ "params": {
+ "warmup_min_lr": "auto",
+ "warmup_max_lr": "auto",
+ "warmup_num_steps": "auto"
+ }
+ },
+
"zero_optimization": {
"stage": 2,
"allgather_partitions": true,
@@ -1041,6 +1151,30 @@ Here is a full ZeRO-2 all-enabled configuration file ``ds_config_zero2.json``:
"cpu_offload": true
},
+ "gradient_accumulation_steps": "auto",
+ "gradient_clipping": "auto",
+ "steps_per_print": 2000,
+ "train_batch_size": "auto",
+ "train_micro_batch_size_per_gpu": "auto",
+ "wall_clock_breakdown": false
+ }
+
+
+Here is a full ZeRO-2 all-enabled manually set configuration file. It is here mainly for you to see what the typical
+values look like, but we highly recommend using the one with multiple ``auto`` settings in it.
+
+.. code-block:: json
+
+ {
+ "fp16": {
+ "enabled": true,
+ "loss_scale": 0,
+ "loss_scale_window": 1000,
+ "initial_scale_power": 16,
+ "hysteresis": 2,
+ "min_loss_scale": 1
+ },
+
"optimizer": {
"type": "AdamW",
"params": {
@@ -1060,6 +1194,17 @@ Here is a full ZeRO-2 all-enabled configuration file ``ds_config_zero2.json``:
}
},
+ "zero_optimization": {
+ "stage": 2,
+ "allgather_partitions": true,
+ "allgather_bucket_size": 2e8,
+ "overlap_comm": true,
+ "reduce_scatter": true,
+ "reduce_bucket_size": 2e8,
+ "contiguous_gradients": true,
+ "cpu_offload": true
+ },
+
"steps_per_print": 2000,
"wall_clock_breakdown": false
}
@@ -1069,13 +1214,14 @@ Here is a full ZeRO-2 all-enabled configuration file ``ds_config_zero2.json``:
ZeRO-3 Example
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-Here is a full ZeRO-3 all-enabled configuration file ``ds_config_zero3.json``:
+Here is a full ZeRO-3 auto-configuration file ``ds_config_zero3.json``:
+
.. code-block:: json
{
"fp16": {
- "enabled": true,
+ "enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
@@ -1083,22 +1229,69 @@ Here is a full ZeRO-3 all-enabled configuration file ``ds_config_zero3.json``:
"min_loss_scale": 1
},
+ "optimizer": {
+ "type": "AdamW",
+ "params": {
+ "lr": "auto",
+ "betas": "auto",
+ "eps": "auto",
+ "weight_decay": "auto"
+ }
+ },
+
+ "scheduler": {
+ "type": "WarmupLR",
+ "params": {
+ "warmup_min_lr": "auto",
+ "warmup_max_lr": "auto",
+ "warmup_num_steps": "auto"
+ }
+ },
+
"zero_optimization": {
"stage": 3,
- "cpu_offload": true,
- "cpu_offload_params": true,
- "cpu_offload_use_pin_memory" : true,
+ "offload_optimizer": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "offload_param": {
+ "device": "cpu",
+ "pin_memory": true
+ },
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1e14,
- "reduce_bucket_size": 1e6,
- "stage3_prefetch_bucket_size": 0.94e6,
- "stage3_param_persistence_threshold": 1e4,
+ "reduce_bucket_size": "auto",
+ "stage3_prefetch_bucket_size": "auto",
+ "stage3_param_persistence_threshold": "auto",
"stage3_max_live_parameters": 1e9,
"stage3_max_reuse_distance": 1e9,
"stage3_gather_fp16_weights_on_model_save": true
},
+ "gradient_accumulation_steps": "auto",
+ "gradient_clipping": "auto",
+ "steps_per_print": 2000,
+ "train_batch_size": "auto",
+ "train_micro_batch_size_per_gpu": "auto",
+ "wall_clock_breakdown": false
+ }
+
+Here is a full ZeRO-3 all-enabled manually set configuration file. It is here mainly for you to see what the typical
+values look like, but we highly recommend using the one with multiple ``auto`` settings in it.
+
+.. code-block:: json
+
+ {
+ "fp16": {
+ "enabled": true,
+ "loss_scale": 0,
+ "loss_scale_window": 1000,
+ "initial_scale_power": 16,
+ "hysteresis": 2,
+ "min_loss_scale": 1
+ },
+
"optimizer": {
"type": "AdamW",
"params": {
@@ -1118,6 +1311,27 @@ Here is a full ZeRO-3 all-enabled configuration file ``ds_config_zero3.json``:
}
},
+ "zero_optimization": {
+ "stage": 3,
+ "offload_optimizer": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "offload_param": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "overlap_comm": true,
+ "contiguous_gradients": true,
+ "sub_group_size": 1e14,
+ "reduce_bucket_size": 1e6,
+ "stage3_prefetch_bucket_size": 0.94e6,
+ "stage3_param_persistence_threshold": 1e4,
+ "stage3_max_live_parameters": 1e9,
+ "stage3_max_reuse_distance": 1e9,
+ "stage3_gather_fp16_weights_on_model_save": true
+ },
+
"steps_per_print": 2000,
"wall_clock_breakdown": false
}
@@ -1153,7 +1367,7 @@ If you don't configure the ``optimizer`` entry in the configuration file, the :c
automatically set it to ``AdamW`` and will use the supplied values or the defaults for the following command line
arguments: ``--learning_rate``, ``--adam_beta1``, ``--adam_beta2``, ``--adam_epsilon`` and ``--weight_decay``.
-Here is an example of the pre-configured ``optimizer`` entry for ``AdamW``:
+Here is an example of the auto-configured ``optimizer`` entry for ``AdamW``:
.. code-block:: json
@@ -1161,15 +1375,16 @@ Here is an example of the pre-configured ``optimizer`` entry for ``AdamW``:
"optimizer": {
"type": "AdamW",
"params": {
- "lr": 0.001,
- "betas": [0.8, 0.999],
- "eps": 1e-8,
- "weight_decay": 3e-7
+ "lr": "auto",
+ "betas": "auto",
+ "eps": "auto",
+ "weight_decay": "auto"
}
- }
+ }
}
-Note that the command line arguments will override the values in the configuration file. This is so that there is one
+
+Note that the command line arguments will set the values in the configuration file. This is so that there is one
definitive source of the values and to avoid hard to find errors when for example, the learning rate is set to
different values in different places. Command line rules. The values that get overridden are:
@@ -1180,19 +1395,42 @@ different values in different places. Command line rules. The values that get ov
Therefore please remember to tune the shared hyperparameters on the command line.
-If you want to use another optimizer which is not listed above, you will have to add ``"zero_allow_untested_optimizer":
-true`` to the top level configuration.
+You can also set the values explicitly:
+
+.. code-block:: json
+
+ {
+ "optimizer": {
+ "type": "AdamW",
+ "params": {
+ "lr": 0.001,
+ "betas": [0.8, 0.999],
+ "eps": 1e-8,
+ "weight_decay": 3e-7
+ }
+ }
+ }
+
+But then you're on your own synchronizing the :class:`~transformers.Trainer` command line arguments and the DeepSpeed
+configuration.
+
+If you want to use another optimizer which is not listed above, you will have to add to the top level configuration.
-If you want to use one of the officially supported optimizers, configure them explicitly in the configuration file, and
-make sure to adjust the values. e.g. if use Adam you will want ``weight_decay`` around ``0.01``.
+.. code-block:: json
+
+ {
+ "zero_allow_untested_optimizer": true
+ }
+
+Similarly to ``AdamW``, you can configure other officially supported optimizers. Just remember that may have different
+config values. e.g. for Adam you will want ``weight_decay`` around ``0.01``.
Scheduler
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-DeepSpeed supports LRRangeTest, OneCycle, WarmupLR and WarmupDecayLR LR schedulers. The full documentation is `here
-`__.
-
+DeepSpeed supports ``LRRangeTest``, ``OneCycle``, ``WarmupLR`` and ``WarmupDecayLR`` learning rate schedulers. The full
+documentation is `here `__.
Here is where the schedulers overlap between 🤗 Transformers and DeepSpeed:
@@ -1200,12 +1438,11 @@ Here is where the schedulers overlap between 🤗 Transformers and DeepSpeed:
* ``WarmupDecayLR`` via ``--lr_scheduler_type linear``. This is also the default value for ``--lr_scheduler_type``,
therefore, if you don't configure the scheduler this is scheduler that will get configured by default.
-
If you don't configure the ``scheduler`` entry in the configuration file, the :class:`~transformers.Trainer` will use
the values of ``--lr_scheduler_type``, ``--learning_rate`` and ``--warmup_steps`` to configure a 🤗 Transformers version
of it.
-Here is an example of the pre-configured ``scheduler`` entry for ``WarmupLR``:
+Here is an example of the auto-configured ``scheduler`` entry for ``WarmupLR``:
.. code-block:: json
@@ -1213,24 +1450,41 @@ Here is an example of the pre-configured ``scheduler`` entry for ``WarmupLR``:
"scheduler": {
"type": "WarmupLR",
"params": {
- "warmup_min_lr": 0,
- "warmup_max_lr": 0.001,
- "warmup_num_steps": 1000
+ "warmup_min_lr": "auto",
+ "warmup_max_lr": "auto",
+ "warmup_num_steps": "auto"
}
}
}
-Note that the command line arguments will override the values in the configuration file. This is so that there is one
-definitive source of the values and to avoid hard to find errors when for example, the learning rate is set to
-different values in different places. Command line rules. The values that get overridden are:
+Since `"auto"` is used the :class:`~transformers.Trainer` arguments will set the correct values in the configuration
+file. This is so that there is one definitive source of the values and to avoid hard to find errors when, for example,
+the learning rate is set to different values in different places. Command line rules. The values that get set are:
+- ``warmup_min_lr`` with the value of ``0``
- ``warmup_max_lr`` with the value of ``--learning_rate``
- ``warmup_num_steps`` with the value of ``--warmup_steps``
- ``total_num_steps`` with either the value of ``--max_steps`` or if it is not provided, derived automatically at run
time based on the environment and the size of the dataset and other command line arguments (needed for
``WarmupDecayLR``).
-Therefore please remember to tune the shared hyperparameters on the command line.
+You can, of course, take over any or all of the configuration values and set those yourself:
+
+.. code-block:: json
+
+ {
+ "scheduler": {
+ "type": "WarmupLR",
+ "params": {
+ "warmup_min_lr": 0,
+ "warmup_max_lr": 0.001,
+ "warmup_num_steps": 1000
+ }
+ }
+ }
+
+But then you're on your own synchronizing the :class:`~transformers.Trainer` command line arguments and the DeepSpeed
+configuration.
For example, for ``WarmupDecayLR``, you can use the following entry:
@@ -1240,16 +1494,16 @@ For example, for ``WarmupDecayLR``, you can use the following entry:
"scheduler": {
"type": "WarmupDecayLR",
"params": {
- "total_num_steps": 10,
"last_batch_iteration": -1,
- "warmup_min_lr": 0,
- "warmup_max_lr": 0.001,
- "warmup_num_steps": 1000
+ "total_num_steps": "auto",
+ "warmup_min_lr": "auto",
+ "warmup_max_lr": "auto",
+ "warmup_num_steps": "auto"
}
}
}
-and ``warmup_max_lr``, ``warmup_num_steps`` and ``total_num_steps`` will be corrected at loading time.
+and ``total_num_steps`, ``warmup_max_lr``, ``warmup_num_steps`` and ``total_num_steps`` will be set at loading time.
@@ -1258,10 +1512,32 @@ Automatic Mixed Precision
You can use automatic mixed precision with either a pytorch-like AMP way or the apex-like way:
-If you want to use an equivalent of the Pytorch native amp, you can either configure the ``fp16`` entry in the
-configuration file, or use the following command line arguments: ``--fp16 --fp16_backend amp``.
+To configure pytorch AMP-like mode set:
+
+.. code-block:: json
+
+ {
+ "fp16": {
+ "enabled": "auto",
+ "loss_scale": 0,
+ "loss_scale_window": 1000,
+ "initial_scale_power": 16,
+ "hysteresis": 2,
+ "min_loss_scale": 1
+ }
+ }
+
+and the :class:`~transformers.Trainer` will automatically enable or disable it based on the value of
+``args.fp16_backend``. The rest of config values are up to you.
-Here is an example of the ``fp16`` configuration:
+This mode gets enabled when ``--fp16 --fp16_backend amp`` command line args are passed.
+
+.. note::
+
+ At the moment DeepSpeed doesn't supported fp32 mode, though it will become available soon. Until then it will be
+ always set to ``true``.
+
+You can also enable/disable this mode explicitly:
.. code-block:: json
@@ -1270,17 +1546,32 @@ Here is an example of the ``fp16`` configuration:
"enabled": true,
"loss_scale": 0,
"loss_scale_window": 1000,
+ "initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
- },
+ }
}
+But then you're on your own synchronizing the :class:`~transformers.Trainer` command line arguments and the DeepSpeed
+configuration.
+
Here is the `documentation `__.
-If you want to use NVIDIA's apex instead, you can can either configure the ``amp`` entry in the configuration file, or
-use the following command line arguments: ``--fp16 --fp16_backend apex --fp16_opt_level 01``.
+To configure apex AMP-like mode set:
-Here is an example of the ``amp`` configuration:
+.. code-block:: json
+
+ "amp": {
+ "enabled": "auto",
+ "opt_level": "auto"
+ }
+
+and the :class:`~transformers.Trainer` will automatically configure it based on the values of ``args.fp16_backend`` and
+``args.fp16_opt_level``.
+
+This mode gets enabled when ``--fp16 --fp16_backend apex --fp16_opt_level 01`` command line args are passed.
+
+You can also configure this mode explicitly:
.. code-block:: json
@@ -1291,6 +1582,9 @@ Here is an example of the ``amp`` configuration:
}
}
+But then you're on your own synchronizing the :class:`~transformers.Trainer` command line arguments and the DeepSpeed
+configuration.
+
Here is the `documentation
`__.
@@ -1298,43 +1592,55 @@ Here is the `documentation
Gradient Accumulation
=======================================================================================================================
-While normally DeepSpeed gets gradient accumulation configured with:
+To configure gradient accumulation set:
.. code-block:: json
{
- "gradient_accumulation_steps": 3,
+ "gradient_accumulation_steps": "auto"
}
-in this case, to enable gradient accumulation, pass the command line ``--gradient_accumulation_steps 3`` argument as
-normal and it will get injected into the DeepSpeed configuration.
-
-If you try to add it directly to the configuration file, you will receive an error from the ``Trainer`` - this is
-because this setting is needed by the ``Trainer`` too, and so this approach ensures that there is a single way of
-setting this value and thus avoid potential subtle errors.
+and the :class:`~transformers.Trainer` will automatically set it to the value of ``args.gradient_accumulation_steps``.
+You can also set the value explicitly:
+.. code-block:: json
+ {
+ "gradient_accumulation_steps": 3
+ }
+But then you're on your own synchronizing the :class:`~transformers.Trainer` command line arguments and the DeepSpeed
+configuration.
Gradient Clipping
=======================================================================================================================
-If you don't configure the ``gradient_clipping`` entry in the configuration file, the :class:`~transformers.Trainer`
-will use the value of the ``--max_grad_norm`` command line argument to set it.
+To configure gradient gradient clipping set:
+
+.. code-block:: json
+
+ {
+ "gradient_clipping": "auto"
+ }
+
+and the :class:`~transformers.Trainer` will automatically set it to the value of ``args.max_grad_norm``.
-Here is an example of the ``gradient_clipping`` configuration:
+You can also set the value explicitly:
.. code-block:: json
{
- "gradient_clipping": 1.0,
+ "gradient_clipping": 1.0
}
+But then you're on your own synchronizing the :class:`~transformers.Trainer` command line arguments and the DeepSpeed
+configuration.
-Getting the model weights out
+
+Getting The Model Weights Out
=======================================================================================================================
As long as you continue training and resuming using DeepSpeed you don't need to worry about anything. DeepSpeed stores
@@ -1352,6 +1658,16 @@ version of the weights. If this setting is ``False`` ``pytorch_model.bin`` won't
DeepSpeed's ``state_dict`` contains a placeholder and not the real weights. If we were to save this ``state_dict`` it
won't be possible to load it back.
+
+.. code-block:: json
+
+ {
+ "zero_optimization": {
+ "stage3_gather_fp16_weights_on_model_save": true
+ }
+ }
+
+
**FP32 Weights:**
While the fp16 weights are fine for resuming training, if you finished finetuning your model and want to upload it to
@@ -1398,44 +1714,18 @@ This is it. ``pytorch_model.bin`` will now contain the full fp32 model weights c
Note: currently the script requires 2x general RAM of the final fp32 model weights.
-ZeRO 3 Nuances
+
+ZeRO-3 and Infinity Nuances
=======================================================================================================================
-ZeRO 3 is quite different from ZeRO 2 because of its param sharding feature.
+ZeRO-3 is quite different from ZeRO-2 because of its param sharding feature.
+
+ZeRO-Infinity further extends ZeRO-3 to support NVMe memory and multiple other speed and scalability improvements.
While all the efforts were made for things to just work without needing any special changes to your models, in certain
circumstances you may find the following information to be needed.
-Registering External Parameters
-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-
-If layer A needs to access weights belonging to layer B, currently layer A needs to tell DeepSpeed about it. This is
-done with the help of ``deepspeed.zero.register_external_parameter`` that needs to be called in ``A.__init__`` and can
-be seen in the following example:
-
-.. code-block:: python
-
- class ModuleZ3(torch.nn.Module):
- def __init__(self, *args):
- super().__init__(self, *args)
- self.layer1 = SomeLayer()
- self.layer2 = OtherLayer()
- deepspeed.zero.register_external_parameter(self, self.layer1.weight)
-
- def forward(self, input):
- x = self.layer1(input)
- # self.layer1.weight is needed in ModuleZ3.forward
- y = self.layer2(x, self.layer1.weight)
- return y
-
-In general ``transformers`` models don't use this style of referring to other layer's weights so most likely you won't
-need to use it.
-
-For full details on this method please refer to `Registering External Parameters
-`__.
-
-
Constructing Massive Models
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@@ -1455,18 +1745,20 @@ context manager (which is also a function decorator), like so:
As you can see this gives you a randomly initialized model.
If you want to use a pretrained model, ``model_class.from_pretrained`` will activate this feature as long as
-``is_deepspeed_zero3_enabled()`` returns ``True``, which can be set manually via ``deepspeed_zero3_enable(True)``.
-Therefore to enable this feature here is the required sequence:
+``is_deepspeed_zero3_enabled()`` returns ``True``, which currently is setup by the
+class:`~transformers.TrainingArguments` object if the passed DeepSpeed configuration file contains ZeRO-3 config
+section. Thus you must create the :class:`~transformers.TrainingArguments` object **before** calling
+``from_pretrained``. Here is an example of a possible sequence:
.. code-block:: python
- from transformers.integrations import deepspeed_zero3_enable
- deepspeed_zero3_enable(True)
- model = T5ForConditionalGeneration.from_pretrained("t5-small")
+ from transformers import AutoModel, Trainer, TrainingArguments
+ training_args = TrainingArguments(..., deepspeed=ds_config)
+ model = AutoModel.from_pretrained("t5-small")
+ trainer = Trainer(model=model, args=training_args, ...)
-If you're using ``Trainer`` command line arguments which include ``--deepspeed ds_config.json`` with ZeRO-3 config
-enabled, then you can skip ``deepspeed_zero3_enable(True)`` as it will try to discover whether it'll be run under
-ZeRO-3 and ``from_pretrained`` will automatically activate this feature.
+If you're using the official example scripts and your command line arguments include ``--deepspeed ds_config.json``
+with ZeRO-3 config enabled, then everything is already done for you, since this is how example scripts are written.
Note: If the fp16 weights of the model can't fit onto the memory of a single GPU this feature must be used.
@@ -1475,8 +1767,6 @@ For full details on this method and other related features please refer to `Cons
-
-
Gathering Parameters
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@@ -1501,8 +1791,6 @@ larger multi-dimensional shape, this means that the parameter is partitioned and
-
-
Notes
=======================================================================================================================
@@ -1514,6 +1802,7 @@ Notes
with your own trainer, and you will have to adapt the latter according to `the DeepSpeed integration instructions
`__.
+
Main DeepSpeed Resources
=======================================================================================================================
@@ -1526,6 +1815,7 @@ Papers:
- `ZeRO: Memory Optimizations Toward Training Trillion Parameter Models `__
- `ZeRO-Offload: Democratizing Billion-Scale Model Training `__
+- `ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning `__
Finally, please, remember that, HuggingFace :class:`~transformers.Trainer` only integrates DeepSpeed, therefore if you
have any problems or questions with regards to DeepSpeed usage, please, file an issue with `DeepSpeed GitHub
diff --git a/docs/source/migration.md b/docs/source/migration.md
index e44af20c0f1e..7b97867e33e4 100644
--- a/docs/source/migration.md
+++ b/docs/source/migration.md
@@ -169,8 +169,8 @@ Regarding the `TFTrainer` class:
- The `TFTrainer` method `_setup_wandb` is deprecated in favor of `setup_wandb`.
- The `TFTrainer` method `_run_model` is deprecated in favor of `run_model`.
-Regarding the `TrainerArgument` class:
-- The `TrainerArgument` argument `evaluate_during_training` is deprecated in favor of `evaluation_strategy`.
+Regarding the `TrainingArguments` class:
+- The `TrainingArguments` argument `evaluate_during_training` is deprecated in favor of `evaluation_strategy`.
Regarding the Transfo-XL model:
- The Transfo-XL configuration attribute `tie_weight` becomes `tie_words_embeddings`.
diff --git a/docs/source/quicktour.rst b/docs/source/quicktour.rst
index 51d962b79b42..c77da9894c9e 100644
--- a/docs/source/quicktour.rst
+++ b/docs/source/quicktour.rst
@@ -238,23 +238,22 @@ keys directly to tensors, for a PyTorch model, you need to unpack the dictionary
>>> ## TENSORFLOW CODE
>>> tf_outputs = tf_model(tf_batch)
-In 🤗 Transformers, all outputs are tuples (with only one element potentially). Here, we get a tuple with just the final
-activations of the model.
+In 🤗 Transformers, all outputs are objects that contain the model's final activations along with other metadata. These
+objects are described in greater detail :doc:`here `. For now, let's inspect the output ourselves:
.. code-block::
>>> ## PYTORCH CODE
>>> print(pt_outputs)
- (tensor([[-4.0833, 4.3364],
- [ 0.0818, -0.0418]], grad_fn=),)
+ SequenceClassifierOutput(loss=None, logits=tensor([[-4.0833, 4.3364],
+ [ 0.0818, -0.0418]], grad_fn=), hidden_states=None, attentions=None)
>>> ## TENSORFLOW CODE
>>> print(tf_outputs)
- (,)
+ TFSequenceClassifierOutput(loss=None, logits=, hidden_states=None, attentions=None)
-The model can return more than just the final activations, which is why the output is a tuple. Here we only asked for
-the final activations, so we get a tuple with one element.
+Notice how the output object has a ``logits`` attribute. You can use this to access the model's final activations.
.. note::
@@ -267,10 +266,10 @@ Let's apply the SoftMax activation to get predictions.
>>> ## PYTORCH CODE
>>> import torch.nn.functional as F
- >>> pt_predictions = F.softmax(pt_outputs[0], dim=-1)
+ >>> pt_predictions = F.softmax(pt_outputs.logits, dim=-1)
>>> ## TENSORFLOW CODE
>>> import tensorflow as tf
- >>> tf_predictions = tf.nn.softmax(tf_outputs[0], axis=-1)
+ >>> tf.nn.softmax(tf_outputs.logits, axis=-1)
We can see we get the numbers from before:
@@ -286,16 +285,24 @@ We can see we get the numbers from before:
tensor([[2.2043e-04, 9.9978e-01],
[5.3086e-01, 4.6914e-01]], grad_fn=)
-If you have labels, you can provide them to the model, it will return a tuple with the loss and the final activations.
+If you provide the model with labels in addition to inputs, the model output object will also contain a ``loss``
+attribute:
.. code-block::
>>> ## PYTORCH CODE
>>> import torch
>>> pt_outputs = pt_model(**pt_batch, labels = torch.tensor([1, 0]))
+ >>> print(pt_outputs)
+ SequenceClassifierOutput(loss=tensor(0.3167, grad_fn=), logits=tensor([[-4.0833, 4.3364],
+ [ 0.0818, -0.0418]], grad_fn=), hidden_states=None, attentions=None)
>>> ## TENSORFLOW CODE
>>> import tensorflow as tf
>>> tf_outputs = tf_model(tf_batch, labels = tf.constant([1, 0]))
+ >>> print(tf_outputs)
+ TFSequenceClassifierOutput(loss=, logits=, hidden_states=None, attentions=None)
Models are standard `torch.nn.Module `__ or `tf.keras.Model
`__ so you can use them in your usual training loop. 🤗
@@ -323,6 +330,7 @@ loading a saved PyTorch model in a TensorFlow model, use :func:`~transformers.TF
.. code-block::
+ from transformers import TFAutoModel
tokenizer = AutoTokenizer.from_pretrained(save_directory)
model = TFAutoModel.from_pretrained(save_directory, from_pt=True)
@@ -330,6 +338,7 @@ and if you are loading a saved TensorFlow model in a PyTorch model, you should u
.. code-block::
+ from transformers import AutoModel
tokenizer = AutoTokenizer.from_pretrained(save_directory)
model = AutoModel.from_pretrained(save_directory, from_tf=True)
@@ -340,10 +349,12 @@ Lastly, you can also ask the model to return all hidden states and all attention
>>> ## PYTORCH CODE
>>> pt_outputs = pt_model(**pt_batch, output_hidden_states=True, output_attentions=True)
- >>> all_hidden_states, all_attentions = pt_outputs[-2:]
+ >>> all_hidden_states = pt_outputs.hidden_states
+ >>> all_attentions = pt_outputs.attentions
>>> ## TENSORFLOW CODE
>>> tf_outputs = tf_model(tf_batch, output_hidden_states=True, output_attentions=True)
- >>> all_hidden_states, all_attentions = tf_outputs[-2:]
+ >>> all_hidden_states = tf_outputs.hidden_states
+ >>> all_attentions = tf_outputs.attentions
Accessing the code
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -376,16 +387,16 @@ directly instantiate model and tokenizer without the auto magic:
Customizing the model
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-If you want to change how the model itself is built, you can define your custom configuration class. Each architecture
-comes with its own relevant configuration (in the case of DistilBERT, :class:`~transformers.DistilBertConfig`) which
-allows you to specify any of the hidden dimension, dropout rate, etc. If you do core modifications, like changing the
-hidden size, you won't be able to use a pretrained model anymore and will need to train from scratch. You would then
-instantiate the model directly from this configuration.
+If you want to change how the model itself is built, you can define a custom configuration class. Each architecture
+comes with its own relevant configuration. For example, :class:`~transformers.DistilBertConfig` allows you to specify
+parameters such as the hidden dimension, dropout rate, etc for DistilBERT. If you do core modifications, like changing
+the hidden size, you won't be able to use a pretrained model anymore and will need to train from scratch. You would
+then instantiate the model directly from this configuration.
-Here we use the predefined vocabulary of DistilBERT (hence load the tokenizer with the
-:func:`~transformers.DistilBertTokenizer.from_pretrained` method) and initialize the model from scratch (hence
-instantiate the model from the configuration instead of using the
-:func:`~transformers.DistilBertForSequenceClassification.from_pretrained` method).
+Below, we load a predefined vocabulary for a tokenizer with the
+:func:`~transformers.DistilBertTokenizer.from_pretrained` method. However, unlike the tokenizer, we wish to initialize
+the model from scratch. Therefore, we instantiate the model from a configuration instead of using the
+:func:`~transformers.DistilBertForSequenceClassification.from_pretrained` method.
.. code-block::
@@ -402,9 +413,9 @@ instantiate the model from the configuration instead of using the
For something that only changes the head of the model (for instance, the number of labels), you can still use a
pretrained model for the body. For instance, let's define a classifier for 10 different labels using a pretrained body.
-We could create a configuration with all the default values and just change the number of labels, but more easily, you
-can directly pass any argument a configuration would take to the :func:`from_pretrained` method and it will update the
-default configuration with it:
+Instead of creating a new configuration with all the default values just to change the number of labels, we can instead
+pass any argument a configuration would take to the :func:`from_pretrained` method and it will update the default
+configuration appropriately:
.. code-block::
diff --git a/examples/legacy/question-answering/run_squad.py b/examples/legacy/question-answering/run_squad.py
index 84986eff6fec..fd50bf06b770 100644
--- a/examples/legacy/question-answering/run_squad.py
+++ b/examples/legacy/question-answering/run_squad.py
@@ -74,7 +74,7 @@ def to_list(tensor):
def train(args, train_dataset, model, tokenizer):
- """ Train the model """
+ """Train the model"""
if args.local_rank in [-1, 0]:
tb_writer = SummaryWriter()
diff --git a/examples/legacy/run_openai_gpt.py b/examples/legacy/run_openai_gpt.py
index 72314b5edb32..1c0c189420c1 100755
--- a/examples/legacy/run_openai_gpt.py
+++ b/examples/legacy/run_openai_gpt.py
@@ -61,7 +61,7 @@ def accuracy(out, labels):
def load_rocstories_dataset(dataset_path):
- """ Output a list of tuples(story, 1st continuation, 2nd continuation, label) """
+ """Output a list of tuples(story, 1st continuation, 2nd continuation, label)"""
with open(dataset_path, encoding="utf_8") as f:
f = csv.reader(f)
output = []
@@ -184,7 +184,7 @@ def main():
# Load and encode the datasets
def tokenize_and_encode(obj):
- """ Tokenize and encode a nested object """
+ """Tokenize and encode a nested object"""
if isinstance(obj, str):
return tokenizer.convert_tokens_to_ids(tokenizer.tokenize(obj))
elif isinstance(obj, int):
diff --git a/examples/legacy/run_swag.py b/examples/legacy/run_swag.py
index ddce4d20e252..666c1becb3f3 100755
--- a/examples/legacy/run_swag.py
+++ b/examples/legacy/run_swag.py
@@ -276,7 +276,7 @@ def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=Fal
def train(args, train_dataset, model, tokenizer):
- """ Train the model """
+ """Train the model"""
if args.local_rank in [-1, 0]:
tb_writer = SummaryWriter()
diff --git a/examples/legacy/seq2seq/minify_dataset.py b/examples/legacy/seq2seq/minify_dataset.py
index 8fd03196a0bc..e6095cecc8e9 100755
--- a/examples/legacy/seq2seq/minify_dataset.py
+++ b/examples/legacy/seq2seq/minify_dataset.py
@@ -19,7 +19,7 @@
def minify(src_dir: str, dest_dir: str, n: int):
- """Write first n lines of each file f in src_dir to dest_dir/f """
+ """Write first n lines of each file f in src_dir to dest_dir/f"""
src_dir = Path(src_dir)
dest_dir = Path(dest_dir)
dest_dir.mkdir(exist_ok=True)
diff --git a/examples/pytorch/README.md b/examples/pytorch/README.md
index c01ba6749db6..7fb888b27ae0 100644
--- a/examples/pytorch/README.md
+++ b/examples/pytorch/README.md
@@ -50,8 +50,8 @@ For example here is how to truncate all three splits to just 50 samples each:
```
examples/pytorch/token-classification/run_ner.py \
--max_train_samples 50 \
---max_val_samples 50 \
---max_test_samples 50 \
+--max_eval_samples 50 \
+--max_predict_samples 50 \
[...]
```
@@ -119,7 +119,7 @@ When using PyTorch, we support TPUs thanks to `pytorch/xla`. For more context an
very detailed [pytorch/xla README](https://github.com/pytorch/xla/blob/master/README.md).
In this repo, we provide a very simple launcher script named
-[xla_spawn.py](https://github.com/huggingface/transformers/tree/master/examples/xla_spawn.py) that lets you run our
+[xla_spawn.py](https://github.com/huggingface/transformers/tree/master/examples/pytorch/xla_spawn.py) that lets you run our
example scripts on multiple TPU cores without any boilerplate. Just pass a `--num_cores` flag to this script, then your
regular training script with its arguments (this is similar to the `torch.distributed.launch` helper for
`torch.distributed`):
diff --git a/examples/pytorch/language-modeling/run_clm.py b/examples/pytorch/language-modeling/run_clm.py
index ea4b4f0934b0..ad9acaf19639 100755
--- a/examples/pytorch/language-modeling/run_clm.py
+++ b/examples/pytorch/language-modeling/run_clm.py
@@ -126,10 +126,10 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
@@ -397,8 +397,8 @@ def group_texts(examples):
if "validation" not in tokenized_datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = lm_datasets["validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
# Initialize our Trainer
trainer = Trainer(
@@ -439,8 +439,8 @@ def group_texts(examples):
metrics = trainer.evaluate()
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
perplexity = math.exp(metrics["eval_loss"])
metrics["perplexity"] = perplexity
diff --git a/examples/pytorch/language-modeling/run_mlm.py b/examples/pytorch/language-modeling/run_mlm.py
index d0932e5f93fd..f16082ceeb64 100755
--- a/examples/pytorch/language-modeling/run_mlm.py
+++ b/examples/pytorch/language-modeling/run_mlm.py
@@ -157,10 +157,10 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
@@ -419,8 +419,8 @@ def group_texts(examples):
if "validation" not in tokenized_datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = tokenized_datasets["validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
# Data collator
# This one will take care of randomly masking the tokens.
@@ -468,8 +468,8 @@ def group_texts(examples):
metrics = trainer.evaluate()
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
perplexity = math.exp(metrics["eval_loss"])
metrics["perplexity"] = perplexity
diff --git a/examples/pytorch/language-modeling/run_plm.py b/examples/pytorch/language-modeling/run_plm.py
index d7136144729e..5a8be42bf57c 100755
--- a/examples/pytorch/language-modeling/run_plm.py
+++ b/examples/pytorch/language-modeling/run_plm.py
@@ -154,10 +154,10 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
@@ -397,8 +397,8 @@ def group_texts(examples):
if "validation" not in tokenized_datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = tokenized_datasets["validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
# Data collator
data_collator = DataCollatorForPermutationLanguageModeling(
@@ -444,8 +444,8 @@ def group_texts(examples):
metrics = trainer.evaluate()
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
perplexity = math.exp(metrics["eval_loss"])
metrics["perplexity"] = perplexity
diff --git a/examples/pytorch/multiple-choice/run_swag.py b/examples/pytorch/multiple-choice/run_swag.py
index e73e13492f5b..0bc0ded2d87e 100755
--- a/examples/pytorch/multiple-choice/run_swag.py
+++ b/examples/pytorch/multiple-choice/run_swag.py
@@ -127,10 +127,10 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
@@ -363,8 +363,8 @@ def preprocess_function(examples):
if "validation" not in datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = datasets["validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
eval_dataset = eval_dataset.map(
preprocess_function,
batched=True,
@@ -422,8 +422,8 @@ def compute_metrics(eval_predictions):
logger.info("*** Evaluate ***")
metrics = trainer.evaluate()
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
diff --git a/examples/pytorch/question-answering/run_qa.py b/examples/pytorch/question-answering/run_qa.py
index 949456950500..cd1d250c4ae7 100755
--- a/examples/pytorch/question-answering/run_qa.py
+++ b/examples/pytorch/question-answering/run_qa.py
@@ -133,17 +133,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
"value if set."
},
)
@@ -468,9 +468,9 @@ def prepare_validation_features(examples):
if "validation" not in datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_examples = datasets["validation"]
- if data_args.max_val_samples is not None:
+ if data_args.max_eval_samples is not None:
# We will select sample from whole data
- eval_examples = eval_examples.select(range(data_args.max_val_samples))
+ eval_examples = eval_examples.select(range(data_args.max_eval_samples))
# Validation Feature Creation
eval_dataset = eval_examples.map(
prepare_validation_features,
@@ -479,28 +479,28 @@ def prepare_validation_features(examples):
remove_columns=column_names,
load_from_cache_file=not data_args.overwrite_cache,
)
- if data_args.max_val_samples is not None:
+ if data_args.max_eval_samples is not None:
# During Feature creation dataset samples might increase, we will select required samples again
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
if training_args.do_predict:
if "test" not in datasets:
raise ValueError("--do_predict requires a test dataset")
- test_examples = datasets["test"]
- if data_args.max_test_samples is not None:
+ predict_examples = datasets["test"]
+ if data_args.max_predict_samples is not None:
# We will select sample from whole data
- test_examples = test_examples.select(range(data_args.max_test_samples))
- # Test Feature Creation
- test_dataset = test_examples.map(
+ predict_examples = predict_examples.select(range(data_args.max_predict_samples))
+ # Predict Feature Creation
+ predict_dataset = predict_examples.map(
prepare_validation_features,
batched=True,
num_proc=data_args.preprocessing_num_workers,
remove_columns=column_names,
load_from_cache_file=not data_args.overwrite_cache,
)
- if data_args.max_test_samples is not None:
+ if data_args.max_predict_samples is not None:
# During Feature creation dataset samples might increase, we will select required samples again
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
# Data collator
# We have already padded to max length if the corresponding flag is True, otherwise we need to pad in the data
@@ -581,8 +581,8 @@ def compute_metrics(p: EvalPrediction):
logger.info("*** Evaluate ***")
metrics = trainer.evaluate()
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
@@ -590,14 +590,16 @@ def compute_metrics(p: EvalPrediction):
# Prediction
if training_args.do_predict:
logger.info("*** Predict ***")
- results = trainer.predict(test_dataset, test_examples)
+ results = trainer.predict(predict_dataset, predict_examples)
metrics = results.metrics
- max_test_samples = data_args.max_test_samples if data_args.max_test_samples is not None else len(test_dataset)
- metrics["test_samples"] = min(max_test_samples, len(test_dataset))
+ max_predict_samples = (
+ data_args.max_predict_samples if data_args.max_predict_samples is not None else len(predict_dataset)
+ )
+ metrics["predict_samples"] = min(max_predict_samples, len(predict_dataset))
- trainer.log_metrics("test", metrics)
- trainer.save_metrics("test", metrics)
+ trainer.log_metrics("predict", metrics)
+ trainer.save_metrics("predict", metrics)
if training_args.push_to_hub:
trainer.push_to_hub()
diff --git a/examples/pytorch/question-answering/run_qa_beam_search.py b/examples/pytorch/question-answering/run_qa_beam_search.py
index ca8d620fe0e0..ffefee12f7a1 100755
--- a/examples/pytorch/question-answering/run_qa_beam_search.py
+++ b/examples/pytorch/question-answering/run_qa_beam_search.py
@@ -132,17 +132,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
"value if set."
},
)
@@ -504,9 +504,9 @@ def prepare_validation_features(examples):
if "validation" not in datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_examples = datasets["validation"]
- if data_args.max_val_samples is not None:
+ if data_args.max_eval_samples is not None:
# Selecting Eval Samples from Dataset
- eval_examples = eval_examples.select(range(data_args.max_val_samples))
+ eval_examples = eval_examples.select(range(data_args.max_eval_samples))
# Create Features from Eval Dataset
eval_dataset = eval_examples.map(
prepare_validation_features,
@@ -515,28 +515,28 @@ def prepare_validation_features(examples):
remove_columns=column_names,
load_from_cache_file=not data_args.overwrite_cache,
)
- if data_args.max_val_samples is not None:
+ if data_args.max_eval_samples is not None:
# Selecting Samples from Dataset again since Feature Creation might increase samples size
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
if training_args.do_predict:
if "test" not in datasets:
raise ValueError("--do_predict requires a test dataset")
- test_examples = datasets["test"]
- if data_args.max_test_samples is not None:
+ predict_examples = datasets["test"]
+ if data_args.max_predict_samples is not None:
# We will select sample from whole data
- test_examples = test_examples.select(range(data_args.max_test_samples))
+ predict_examples = predict_examples.select(range(data_args.max_predict_samples))
# Test Feature Creation
- test_dataset = test_examples.map(
+ predict_dataset = predict_examples.map(
prepare_validation_features,
batched=True,
num_proc=data_args.preprocessing_num_workers,
remove_columns=column_names,
load_from_cache_file=not data_args.overwrite_cache,
)
- if data_args.max_test_samples is not None:
+ if data_args.max_predict_samples is not None:
# During Feature creation dataset samples might increase, we will select required samples again
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
# Data collator
# We have already padded to max length if the corresponding flag is True, otherwise we need to pad in the data
@@ -620,8 +620,8 @@ def compute_metrics(p: EvalPrediction):
logger.info("*** Evaluate ***")
metrics = trainer.evaluate()
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
@@ -629,14 +629,16 @@ def compute_metrics(p: EvalPrediction):
# Prediction
if training_args.do_predict:
logger.info("*** Predict ***")
- results = trainer.predict(test_dataset, test_examples)
+ results = trainer.predict(predict_dataset, predict_examples)
metrics = results.metrics
- max_test_samples = data_args.max_test_samples if data_args.max_test_samples is not None else len(test_dataset)
- metrics["test_samples"] = min(max_test_samples, len(test_dataset))
+ max_predict_samples = (
+ data_args.max_predict_samples if data_args.max_predict_samples is not None else len(predict_dataset)
+ )
+ metrics["predict_samples"] = min(max_predict_samples, len(predict_dataset))
- trainer.log_metrics("test", metrics)
- trainer.save_metrics("test", metrics)
+ trainer.log_metrics("predict", metrics)
+ trainer.save_metrics("predict", metrics)
if training_args.push_to_hub:
trainer.push_to_hub()
diff --git a/examples/pytorch/question-answering/run_qa_beam_search_no_trainer.py b/examples/pytorch/question-answering/run_qa_beam_search_no_trainer.py
index ca0d60c0f8d1..f1d5a2d03083 100644
--- a/examples/pytorch/question-answering/run_qa_beam_search_no_trainer.py
+++ b/examples/pytorch/question-answering/run_qa_beam_search_no_trainer.py
@@ -183,20 +183,20 @@ def parse_args():
"value if set.",
)
parser.add_argument(
- "--max_val_samples",
+ "--max_eval_samples",
type=int,
default=None,
- help="For debugging purposes or quicker training, truncate the number of validation examples to this "
+ help="For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set.",
)
parser.add_argument(
"--overwrite_cache", type=bool, default=False, help="Overwrite the cached training and evaluation sets"
)
parser.add_argument(
- "--max_test_samples",
+ "--max_predict_samples",
type=int,
default=None,
- help="For debugging purposes or quicker training, truncate the number of test examples to this",
+ help="For debugging purposes or quicker training, truncate the number of prediction examples to this",
)
args = parser.parse_args()
@@ -481,9 +481,9 @@ def prepare_validation_features(examples):
if "validation" not in raw_datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_examples = raw_datasets["validation"]
- if args.max_val_samples is not None:
+ if args.max_eval_samples is not None:
# We will select sample from whole data
- eval_examples = eval_examples.select(range(args.max_val_samples))
+ eval_examples = eval_examples.select(range(args.max_eval_samples))
# Validation Feature Creation
eval_dataset = eval_examples.map(
prepare_validation_features,
@@ -493,28 +493,28 @@ def prepare_validation_features(examples):
load_from_cache_file=not args.overwrite_cache,
)
- if args.max_val_samples is not None:
+ if args.max_eval_samples is not None:
# During Feature creation dataset samples might increase, we will select required samples again
- eval_dataset = eval_dataset.select(range(args.max_val_samples))
+ eval_dataset = eval_dataset.select(range(args.max_eval_samples))
if args.do_predict:
if "test" not in raw_datasets:
raise ValueError("--do_predict requires a test dataset")
- test_examples = raw_datasets["test"]
- if args.max_test_samples is not None:
+ predict_examples = raw_datasets["test"]
+ if args.max_predict_samples is not None:
# We will select sample from whole data
- test_examples = test_examples.select(range(args.max_test_samples))
- # Test Feature Creation
- test_dataset = test_examples.map(
+ predict_examples = predict_examples.select(range(args.max_predict_samples))
+ # Predict Feature Creation
+ predict_dataset = predict_examples.map(
prepare_validation_features,
batched=True,
num_proc=args.preprocessing_num_workers,
remove_columns=column_names,
load_from_cache_file=not args.overwrite_cache,
)
- if args.max_test_samples is not None:
+ if args.max_predict_samples is not None:
# During Feature creation dataset samples might increase, we will select required samples again
- test_dataset = test_dataset.select(range(args.max_test_samples))
+ predict_dataset = predict_dataset.select(range(args.max_predict_samples))
# Log a few random samples from the training set:
for index in random.sample(range(len(train_dataset)), 3):
@@ -539,9 +539,9 @@ def prepare_validation_features(examples):
eval_dataloader = DataLoader(eval_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size)
if args.do_predict:
- test_dataset.set_format(type="torch", columns=["attention_mask", "input_ids", "token_type_ids"])
- test_dataloader = DataLoader(
- test_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size
+ predict_dataset.set_format(type="torch", columns=["attention_mask", "input_ids", "token_type_ids"])
+ predict_dataloader = DataLoader(
+ predict_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size
)
# Post-processing:
@@ -737,7 +737,7 @@ def create_and_fill_np_array(start_or_end_logits, dataset, max_len):
all_end_top_log_probs = []
all_end_top_index = []
all_cls_logits = []
- for step, batch in enumerate(test_dataloader):
+ for step, batch in enumerate(predict_dataloader):
with torch.no_grad():
outputs = model(**batch)
start_top_log_probs = outputs.start_top_log_probs
@@ -762,10 +762,10 @@ def create_and_fill_np_array(start_or_end_logits, dataset, max_len):
max_len = max([x.shape[1] for x in all_end_top_log_probs]) # Get the max_length of the tensor
# concatenate all numpy arrays collected above
- start_top_log_probs_concat = create_and_fill_np_array(all_start_top_log_probs, test_dataset, max_len)
- start_top_index_concat = create_and_fill_np_array(all_start_top_index, test_dataset, max_len)
- end_top_log_probs_concat = create_and_fill_np_array(all_end_top_log_probs, test_dataset, max_len)
- end_top_index_concat = create_and_fill_np_array(all_end_top_index, test_dataset, max_len)
+ start_top_log_probs_concat = create_and_fill_np_array(all_start_top_log_probs, predict_dataset, max_len)
+ start_top_index_concat = create_and_fill_np_array(all_start_top_index, predict_dataset, max_len)
+ end_top_log_probs_concat = create_and_fill_np_array(all_end_top_log_probs, predict_dataset, max_len)
+ end_top_index_concat = create_and_fill_np_array(all_end_top_index, predict_dataset, max_len)
all_cls_logits = np.concatenate(all_cls_logits, axis=0)
# delete the list of numpy arrays
@@ -774,7 +774,7 @@ def create_and_fill_np_array(start_or_end_logits, dataset, max_len):
del end_top_log_probs
del end_top_index
- test_dataset.set_format(type=None, columns=list(test_dataset.features.keys()))
+ predict_dataset.set_format(type=None, columns=list(predict_dataset.features.keys()))
outputs_numpy = (
start_top_log_probs_concat,
start_top_index_concat,
@@ -783,9 +783,9 @@ def create_and_fill_np_array(start_or_end_logits, dataset, max_len):
cls_logits,
)
- prediction = post_processing_function(test_examples, test_dataset, outputs_numpy)
- test_metric = metric.compute(predictions=prediction.predictions, references=prediction.label_ids)
- logger.info(f"Test metrics: {test_metric}")
+ prediction = post_processing_function(predict_examples, predict_dataset, outputs_numpy)
+ predict_metric = metric.compute(predictions=prediction.predictions, references=prediction.label_ids)
+ logger.info(f"Predict metrics: {predict_metric}")
if args.output_dir is not None:
accelerator.wait_for_everyone()
diff --git a/examples/pytorch/question-answering/run_qa_no_trainer.py b/examples/pytorch/question-answering/run_qa_no_trainer.py
index 7a8b2215be75..97e2c8b431d0 100755
--- a/examples/pytorch/question-answering/run_qa_no_trainer.py
+++ b/examples/pytorch/question-answering/run_qa_no_trainer.py
@@ -205,20 +205,20 @@ def parse_args():
"value if set.",
)
parser.add_argument(
- "--max_val_samples",
+ "--max_eval_samples",
type=int,
default=None,
- help="For debugging purposes or quicker training, truncate the number of validation examples to this "
+ help="For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set.",
)
parser.add_argument(
"--overwrite_cache", type=bool, default=False, help="Overwrite the cached training and evaluation sets"
)
parser.add_argument(
- "--max_test_samples",
+ "--max_predict_samples",
type=int,
default=None,
- help="For debugging purposes or quicker training, truncate the number of test examples to this",
+ help="For debugging purposes or quicker training, truncate the number of prediction examples to this",
)
parser.add_argument(
"--model_type",
@@ -486,9 +486,9 @@ def prepare_validation_features(examples):
if "validation" not in raw_datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_examples = raw_datasets["validation"]
- if args.max_val_samples is not None:
+ if args.max_eval_samples is not None:
# We will select sample from whole data
- eval_examples = eval_examples.select(range(args.max_val_samples))
+ eval_examples = eval_examples.select(range(args.max_eval_samples))
# Validation Feature Creation
eval_dataset = eval_examples.map(
prepare_validation_features,
@@ -498,28 +498,28 @@ def prepare_validation_features(examples):
load_from_cache_file=not args.overwrite_cache,
)
- if args.max_val_samples is not None:
+ if args.max_eval_samples is not None:
# During Feature creation dataset samples might increase, we will select required samples again
- eval_dataset = eval_dataset.select(range(args.max_val_samples))
+ eval_dataset = eval_dataset.select(range(args.max_eval_samples))
if args.do_predict:
if "test" not in raw_datasets:
raise ValueError("--do_predict requires a test dataset")
- test_examples = raw_datasets["test"]
- if args.max_test_samples is not None:
+ predict_examples = raw_datasets["test"]
+ if args.max_predict_samples is not None:
# We will select sample from whole data
- test_examples = test_examples.select(range(args.max_test_samples))
- # Test Feature Creation
- test_dataset = test_examples.map(
+ predict_examples = predict_examples.select(range(args.max_predict_samples))
+ # Predict Feature Creation
+ predict_dataset = predict_examples.map(
prepare_validation_features,
batched=True,
num_proc=args.preprocessing_num_workers,
remove_columns=column_names,
load_from_cache_file=not args.overwrite_cache,
)
- if args.max_test_samples is not None:
+ if args.max_predict_samples is not None:
# During Feature creation dataset samples might increase, we will select required samples again
- test_dataset = test_dataset.select(range(args.max_test_samples))
+ predict_dataset = predict_dataset.select(range(args.max_predict_samples))
# Log a few random samples from the training set:
for index in random.sample(range(len(train_dataset)), 3):
@@ -544,9 +544,9 @@ def prepare_validation_features(examples):
eval_dataloader = DataLoader(eval_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size)
if args.do_predict:
- test_dataset.set_format(type="torch", columns=["attention_mask", "input_ids", "token_type_ids"])
- test_dataloader = DataLoader(
- test_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size
+ predict_dataset.set_format(type="torch", columns=["attention_mask", "input_ids", "token_type_ids"])
+ predict_dataloader = DataLoader(
+ predict_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size
)
# Post-processing:
@@ -714,7 +714,7 @@ def create_and_fill_np_array(start_or_end_logits, dataset, max_len):
if args.do_predict:
all_start_logits = []
all_end_logits = []
- for step, batch in enumerate(test_dataloader):
+ for step, batch in enumerate(predict_dataloader):
with torch.no_grad():
outputs = model(**batch)
start_logits = outputs.start_logits
@@ -729,19 +729,19 @@ def create_and_fill_np_array(start_or_end_logits, dataset, max_len):
max_len = max([x.shape[1] for x in all_start_logits]) # Get the max_length of the tensor
# concatenate the numpy array
- start_logits_concat = create_and_fill_np_array(all_start_logits, test_dataset, max_len)
- end_logits_concat = create_and_fill_np_array(all_end_logits, test_dataset, max_len)
+ start_logits_concat = create_and_fill_np_array(all_start_logits, predict_dataset, max_len)
+ end_logits_concat = create_and_fill_np_array(all_end_logits, predict_dataset, max_len)
# delete the list of numpy arrays
del all_start_logits
del all_end_logits
# Now we need to add extra columns which we removed for post processing
- test_dataset.set_format(type=None, columns=list(test_dataset.features.keys()))
+ predict_dataset.set_format(type=None, columns=list(predict_dataset.features.keys()))
outputs_numpy = (start_logits_concat, end_logits_concat)
- prediction = post_processing_function(test_examples, test_dataset, outputs_numpy)
- eval_metric = metric.compute(predictions=prediction.predictions, references=prediction.label_ids)
- logger.info(f"Test metrics: {eval_metric}")
+ prediction = post_processing_function(predict_examples, predict_dataset, outputs_numpy)
+ predict_metric = metric.compute(predictions=prediction.predictions, references=prediction.label_ids)
+ logger.info(f"Predict metrics: {predict_metric}")
if args.output_dir is not None:
accelerator.wait_for_everyone()
diff --git a/examples/pytorch/question-answering/trainer_qa.py b/examples/pytorch/question-answering/trainer_qa.py
index 41699cd1dfae..36e2e544a7ac 100644
--- a/examples/pytorch/question-answering/trainer_qa.py
+++ b/examples/pytorch/question-answering/trainer_qa.py
@@ -66,16 +66,16 @@ def evaluate(self, eval_dataset=None, eval_examples=None, ignore_keys=None):
self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, metrics)
return metrics
- def predict(self, test_dataset, test_examples, ignore_keys=None):
- test_dataloader = self.get_test_dataloader(test_dataset)
+ def predict(self, predict_dataset, predict_examples, ignore_keys=None):
+ predict_dataloader = self.get_test_dataloader(predict_dataset)
# Temporarily disable metric computation, we will do it in the loop here.
compute_metrics = self.compute_metrics
self.compute_metrics = None
try:
output = self.prediction_loop(
- test_dataloader,
- description="Evaluation",
+ predict_dataloader,
+ description="Prediction",
# No point gathering the predictions if there are no metrics, otherwise we defer to
# self.args.prediction_loss_only
prediction_loss_only=True if compute_metrics is None else None,
@@ -87,7 +87,7 @@ def predict(self, test_dataset, test_examples, ignore_keys=None):
if self.post_process_function is None or self.compute_metrics is None:
return output
- eval_preds = self.post_process_function(test_examples, test_dataset, output.predictions, "test")
- metrics = self.compute_metrics(eval_preds)
+ predictions = self.post_process_function(predict_examples, predict_dataset, output.predictions, "predict")
+ metrics = self.compute_metrics(predictions)
- return PredictionOutput(predictions=eval_preds.predictions, label_ids=eval_preds.label_ids, metrics=metrics)
+ return PredictionOutput(predictions=predictions.predictions, label_ids=predictions.label_ids, metrics=metrics)
diff --git a/examples/pytorch/summarization/run_summarization.py b/examples/pytorch/summarization/run_summarization.py
index 25a117010e3e..841d7e9b58a1 100755
--- a/examples/pytorch/summarization/run_summarization.py
+++ b/examples/pytorch/summarization/run_summarization.py
@@ -178,17 +178,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
"value if set."
},
)
@@ -438,8 +438,8 @@ def preprocess_function(examples):
if "validation" not in datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = datasets["validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
eval_dataset = eval_dataset.map(
preprocess_function,
batched=True,
@@ -452,10 +452,10 @@ def preprocess_function(examples):
max_target_length = data_args.val_max_target_length
if "test" not in datasets:
raise ValueError("--do_predict requires a test dataset")
- test_dataset = datasets["test"]
- if data_args.max_test_samples is not None:
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
- test_dataset = test_dataset.map(
+ predict_dataset = datasets["test"]
+ if data_args.max_predict_samples is not None:
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
+ predict_dataset = predict_dataset.map(
preprocess_function,
batched=True,
num_proc=data_args.preprocessing_num_workers,
@@ -547,37 +547,39 @@ def compute_metrics(eval_preds):
metrics = trainer.evaluate(
max_length=data_args.val_max_target_length, num_beams=data_args.num_beams, metric_key_prefix="eval"
)
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
if training_args.do_predict:
- logger.info("*** Test ***")
+ logger.info("*** Predict ***")
- test_results = trainer.predict(
- test_dataset,
- metric_key_prefix="test",
+ predict_results = trainer.predict(
+ predict_dataset,
+ metric_key_prefix="predict",
max_length=data_args.val_max_target_length,
num_beams=data_args.num_beams,
)
- metrics = test_results.metrics
- max_test_samples = data_args.max_test_samples if data_args.max_test_samples is not None else len(test_dataset)
- metrics["test_samples"] = min(max_test_samples, len(test_dataset))
+ metrics = predict_results.metrics
+ max_predict_samples = (
+ data_args.max_predict_samples if data_args.max_predict_samples is not None else len(predict_dataset)
+ )
+ metrics["predict_samples"] = min(max_predict_samples, len(predict_dataset))
- trainer.log_metrics("test", metrics)
- trainer.save_metrics("test", metrics)
+ trainer.log_metrics("predict", metrics)
+ trainer.save_metrics("predict", metrics)
if trainer.is_world_process_zero():
if training_args.predict_with_generate:
- test_preds = tokenizer.batch_decode(
- test_results.predictions, skip_special_tokens=True, clean_up_tokenization_spaces=True
+ predictions = tokenizer.batch_decode(
+ predict_results.predictions, skip_special_tokens=True, clean_up_tokenization_spaces=True
)
- test_preds = [pred.strip() for pred in test_preds]
- output_test_preds_file = os.path.join(training_args.output_dir, "test_generations.txt")
- with open(output_test_preds_file, "w") as writer:
- writer.write("\n".join(test_preds))
+ predictions = [pred.strip() for pred in predictions]
+ output_prediction_file = os.path.join(training_args.output_dir, "generated_predictions.txt")
+ with open(output_prediction_file, "w") as writer:
+ writer.write("\n".join(predictions))
if training_args.push_to_hub:
trainer.push_to_hub()
diff --git a/examples/pytorch/text-classification/run_glue.py b/examples/pytorch/text-classification/run_glue.py
index eff048a65a0b..cd8c6a94aefc 100755
--- a/examples/pytorch/text-classification/run_glue.py
+++ b/examples/pytorch/text-classification/run_glue.py
@@ -100,17 +100,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
"value if set."
},
)
@@ -390,15 +390,15 @@ def preprocess_function(examples):
if "validation" not in datasets and "validation_matched" not in datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = datasets["validation_matched" if data_args.task_name == "mnli" else "validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
if training_args.do_predict or data_args.task_name is not None or data_args.test_file is not None:
if "test" not in datasets and "test_matched" not in datasets:
raise ValueError("--do_predict requires a test dataset")
- test_dataset = datasets["test_matched" if data_args.task_name == "mnli" else "test"]
- if data_args.max_test_samples is not None:
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
+ predict_dataset = datasets["test_matched" if data_args.task_name == "mnli" else "test"]
+ if data_args.max_predict_samples is not None:
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
# Log a few random samples from the training set:
if training_args.do_train:
@@ -483,32 +483,34 @@ def compute_metrics(p: EvalPrediction):
for eval_dataset, task in zip(eval_datasets, tasks):
metrics = trainer.evaluate(eval_dataset=eval_dataset)
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = (
+ data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ )
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
if training_args.do_predict:
- logger.info("*** Test ***")
+ logger.info("*** Predict ***")
# Loop to handle MNLI double evaluation (matched, mis-matched)
tasks = [data_args.task_name]
- test_datasets = [test_dataset]
+ predict_datasets = [predict_dataset]
if data_args.task_name == "mnli":
tasks.append("mnli-mm")
- test_datasets.append(datasets["test_mismatched"])
+ predict_datasets.append(datasets["test_mismatched"])
- for test_dataset, task in zip(test_datasets, tasks):
+ for predict_dataset, task in zip(predict_datasets, tasks):
# Removing the `label` columns because it contains -1 and Trainer won't like that.
- test_dataset.remove_columns_("label")
- predictions = trainer.predict(test_dataset=test_dataset).predictions
+ predict_dataset.remove_columns_("label")
+ predictions = trainer.predict(predict_dataset, metric_key_prefix="predict").predictions
predictions = np.squeeze(predictions) if is_regression else np.argmax(predictions, axis=1)
- output_test_file = os.path.join(training_args.output_dir, f"test_results_{task}.txt")
+ output_predict_file = os.path.join(training_args.output_dir, f"predict_results_{task}.txt")
if trainer.is_world_process_zero():
- with open(output_test_file, "w") as writer:
- logger.info(f"***** Test results {task} *****")
+ with open(output_predict_file, "w") as writer:
+ logger.info(f"***** Predict results {task} *****")
writer.write("index\tprediction\n")
for index, item in enumerate(predictions):
if is_regression:
diff --git a/examples/pytorch/text-classification/run_xnli.py b/examples/pytorch/text-classification/run_xnli.py
index 1c74e6e7bc0f..c1d8522c8d04 100755
--- a/examples/pytorch/text-classification/run_xnli.py
+++ b/examples/pytorch/text-classification/run_xnli.py
@@ -84,17 +84,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
"value if set."
},
)
@@ -221,8 +221,8 @@ def main():
label_list = eval_dataset.features["label"].names
if training_args.do_predict:
- test_dataset = load_dataset("xnli", model_args.language, split="test", cache_dir=model_args.cache_dir)
- label_list = test_dataset.features["label"].names
+ predict_dataset = load_dataset("xnli", model_args.language, split="test", cache_dir=model_args.cache_dir)
+ label_list = predict_dataset.features["label"].names
# Labels
num_labels = len(label_list)
@@ -286,8 +286,8 @@ def preprocess_function(examples):
logger.info(f"Sample {index} of the training set: {train_dataset[index]}.")
if training_args.do_eval:
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
eval_dataset = eval_dataset.map(
preprocess_function,
batched=True,
@@ -295,9 +295,9 @@ def preprocess_function(examples):
)
if training_args.do_predict:
- if data_args.max_test_samples is not None:
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
- test_dataset = test_dataset.map(
+ if data_args.max_predict_samples is not None:
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
+ predict_dataset = predict_dataset.map(
preprocess_function,
batched=True,
load_from_cache_file=not data_args.overwrite_cache,
@@ -360,8 +360,8 @@ def compute_metrics(p: EvalPrediction):
logger.info("*** Evaluate ***")
metrics = trainer.evaluate(eval_dataset=eval_dataset)
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
@@ -369,18 +369,20 @@ def compute_metrics(p: EvalPrediction):
# Prediction
if training_args.do_predict:
logger.info("*** Predict ***")
- predictions, labels, metrics = trainer.predict(test_dataset)
+ predictions, labels, metrics = trainer.predict(predict_dataset, metric_key_prefix="predict")
- max_test_samples = data_args.max_test_samples if data_args.max_test_samples is not None else len(test_dataset)
- metrics["test_samples"] = min(max_test_samples, len(test_dataset))
+ max_predict_samples = (
+ data_args.max_predict_samples if data_args.max_predict_samples is not None else len(predict_dataset)
+ )
+ metrics["predict_samples"] = min(max_predict_samples, len(predict_dataset))
- trainer.log_metrics("test", metrics)
- trainer.save_metrics("test", metrics)
+ trainer.log_metrics("predict", metrics)
+ trainer.save_metrics("predict", metrics)
predictions = np.argmax(predictions, axis=1)
- output_test_file = os.path.join(training_args.output_dir, "test_predictions.txt")
+ output_predict_file = os.path.join(training_args.output_dir, "predictions.txt")
if trainer.is_world_process_zero():
- with open(output_test_file, "w") as writer:
+ with open(output_predict_file, "w") as writer:
writer.write("index\tprediction\n")
for index, item in enumerate(predictions):
item = label_list[item]
diff --git a/examples/pytorch/token-classification/run_ner.py b/examples/pytorch/token-classification/run_ner.py
index 2907a78033b1..6ca2c591aeb4 100755
--- a/examples/pytorch/token-classification/run_ner.py
+++ b/examples/pytorch/token-classification/run_ner.py
@@ -128,17 +128,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
"value if set."
},
)
@@ -363,8 +363,8 @@ def tokenize_and_align_labels(examples):
if "validation" not in datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = datasets["validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
eval_dataset = eval_dataset.map(
tokenize_and_align_labels,
batched=True,
@@ -375,10 +375,10 @@ def tokenize_and_align_labels(examples):
if training_args.do_predict:
if "test" not in datasets:
raise ValueError("--do_predict requires a test dataset")
- test_dataset = datasets["test"]
- if data_args.max_test_samples is not None:
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
- test_dataset = test_dataset.map(
+ predict_dataset = datasets["test"]
+ if data_args.max_predict_samples is not None:
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
+ predict_dataset = predict_dataset.map(
tokenize_and_align_labels,
batched=True,
num_proc=data_args.preprocessing_num_workers,
@@ -462,8 +462,8 @@ def compute_metrics(p):
metrics = trainer.evaluate()
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
@@ -472,7 +472,7 @@ def compute_metrics(p):
if training_args.do_predict:
logger.info("*** Predict ***")
- predictions, labels, metrics = trainer.predict(test_dataset)
+ predictions, labels, metrics = trainer.predict(predict_dataset, metric_key_prefix="predict")
predictions = np.argmax(predictions, axis=2)
# Remove ignored index (special tokens)
@@ -481,13 +481,13 @@ def compute_metrics(p):
for prediction, label in zip(predictions, labels)
]
- trainer.log_metrics("test", metrics)
- trainer.save_metrics("test", metrics)
+ trainer.log_metrics("predict", metrics)
+ trainer.save_metrics("predict", metrics)
# Save predictions
- output_test_predictions_file = os.path.join(training_args.output_dir, "test_predictions.txt")
+ output_predictions_file = os.path.join(training_args.output_dir, "predictions.txt")
if trainer.is_world_process_zero():
- with open(output_test_predictions_file, "w") as writer:
+ with open(output_predictions_file, "w") as writer:
for prediction in true_predictions:
writer.write(" ".join(prediction) + "\n")
diff --git a/examples/pytorch/translation/run_translation.py b/examples/pytorch/translation/run_translation.py
index 24ab358cc30d..e4fd946e716a 100755
--- a/examples/pytorch/translation/run_translation.py
+++ b/examples/pytorch/translation/run_translation.py
@@ -167,17 +167,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
"value if set."
},
)
@@ -432,8 +432,8 @@ def preprocess_function(examples):
if "validation" not in datasets:
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = datasets["validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
eval_dataset = eval_dataset.map(
preprocess_function,
batched=True,
@@ -446,10 +446,10 @@ def preprocess_function(examples):
max_target_length = data_args.val_max_target_length
if "test" not in datasets:
raise ValueError("--do_predict requires a test dataset")
- test_dataset = datasets["test"]
- if data_args.max_test_samples is not None:
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
- test_dataset = test_dataset.map(
+ predict_dataset = datasets["test"]
+ if data_args.max_predict_samples is not None:
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
+ predict_dataset = predict_dataset.map(
preprocess_function,
batched=True,
num_proc=data_args.preprocessing_num_workers,
@@ -539,37 +539,39 @@ def compute_metrics(eval_preds):
metrics = trainer.evaluate(
max_length=data_args.val_max_target_length, num_beams=data_args.num_beams, metric_key_prefix="eval"
)
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
if training_args.do_predict:
- logger.info("*** Test ***")
+ logger.info("*** Predict ***")
- test_results = trainer.predict(
- test_dataset,
- metric_key_prefix="test",
+ predict_results = trainer.predict(
+ predict_dataset,
+ metric_key_prefix="predict",
max_length=data_args.val_max_target_length,
num_beams=data_args.num_beams,
)
- metrics = test_results.metrics
- max_test_samples = data_args.max_test_samples if data_args.max_test_samples is not None else len(test_dataset)
- metrics["test_samples"] = min(max_test_samples, len(test_dataset))
+ metrics = predict_results.metrics
+ max_predict_samples = (
+ data_args.max_predict_samples if data_args.max_predict_samples is not None else len(predict_dataset)
+ )
+ metrics["predict_samples"] = min(max_predict_samples, len(predict_dataset))
- trainer.log_metrics("test", metrics)
- trainer.save_metrics("test", metrics)
+ trainer.log_metrics("predict", metrics)
+ trainer.save_metrics("predict", metrics)
if trainer.is_world_process_zero():
if training_args.predict_with_generate:
- test_preds = tokenizer.batch_decode(
- test_results.predictions, skip_special_tokens=True, clean_up_tokenization_spaces=True
+ predictions = tokenizer.batch_decode(
+ predict_results.predictions, skip_special_tokens=True, clean_up_tokenization_spaces=True
)
- test_preds = [pred.strip() for pred in test_preds]
- output_test_preds_file = os.path.join(training_args.output_dir, "test_generations.txt")
- with open(output_test_preds_file, "w") as writer:
- writer.write("\n".join(test_preds))
+ predictions = [pred.strip() for pred in predictions]
+ output_prediction_file = os.path.join(training_args.output_dir, "generated_predictions.txt")
+ with open(output_prediction_file, "w") as writer:
+ writer.write("\n".join(predictions))
if training_args.push_to_hub:
trainer.push_to_hub()
diff --git a/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py b/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py
index 1ac84f28d381..0366366d7124 100755
--- a/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py
+++ b/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py
@@ -71,7 +71,7 @@ def set_seed(args):
def train(args, train_dataset, model, tokenizer):
- """ Train the model """
+ """Train the model"""
if args.local_rank in [-1, 0]:
tb_writer = SummaryWriter()
diff --git a/examples/research_projects/bertabs/modeling_bertabs.py b/examples/research_projects/bertabs/modeling_bertabs.py
index ce0e25e2b149..a7d8611a265f 100644
--- a/examples/research_projects/bertabs/modeling_bertabs.py
+++ b/examples/research_projects/bertabs/modeling_bertabs.py
@@ -251,7 +251,7 @@ def forward(
return output, state # , state
def init_decoder_state(self, src, memory_bank, with_cache=False):
- """ Init decoder state """
+ """Init decoder state"""
state = TransformerDecoderState(src)
if with_cache:
state._init_cache(memory_bank, self.num_layers)
@@ -479,11 +479,11 @@ def forward(
head_count = self.head_count
def shape(x):
- """ projection """
+ """projection"""
return x.view(batch_size, -1, head_count, dim_per_head).transpose(1, 2)
def unshape(x):
- """ compute context """
+ """compute context"""
return x.transpose(1, 2).contiguous().view(batch_size, -1, head_count * dim_per_head)
# 1) Project key, value, and query.
@@ -571,12 +571,12 @@ class DecoderState(object):
"""
def detach(self):
- """ Need to document this """
+ """Need to document this"""
self.hidden = tuple([_.detach() for _ in self.hidden])
self.input_feed = self.input_feed.detach()
def beam_update(self, idx, positions, beam_size):
- """ Need to document this """
+ """Need to document this"""
for e in self._all:
sizes = e.size()
br = sizes[1]
@@ -592,7 +592,7 @@ def map_batch_fn(self, fn):
class TransformerDecoderState(DecoderState):
- """ Transformer Decoder state base class """
+ """Transformer Decoder state base class"""
def __init__(self, src):
"""
@@ -638,7 +638,7 @@ def _init_cache(self, memory_bank, num_layers):
self.cache["layer_{}".format(l)] = layer_cache
def repeat_beam_size_times(self, beam_size):
- """ Repeat beam_size times along batch dimension. """
+ """Repeat beam_size times along batch dimension."""
self.src = self.src.data.repeat(1, beam_size, 1)
def map_batch_fn(self, fn):
diff --git a/examples/research_projects/bertabs/test_utils_summarization.py b/examples/research_projects/bertabs/test_utils_summarization.py
index 5af089824664..18120c9063ed 100644
--- a/examples/research_projects/bertabs/test_utils_summarization.py
+++ b/examples/research_projects/bertabs/test_utils_summarization.py
@@ -25,19 +25,19 @@ def setUp(self):
self.block_size = 10
def test_fit_to_block_sequence_too_small(self):
- """ Pad the sequence with 0 if the sequence is smaller than the block size."""
+ """Pad the sequence with 0 if the sequence is smaller than the block size."""
sequence = [1, 2, 3, 4]
expected_output = [1, 2, 3, 4, 0, 0, 0, 0, 0, 0]
self.assertEqual(truncate_or_pad(sequence, self.block_size, 0), expected_output)
def test_fit_to_block_sequence_fit_exactly(self):
- """ Do nothing if the sequence is the right size. """
+ """Do nothing if the sequence is the right size."""
sequence = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
expected_output = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
self.assertEqual(truncate_or_pad(sequence, self.block_size, 0), expected_output)
def test_fit_to_block_sequence_too_big(self):
- """ Truncate the sequence if it is too long. """
+ """Truncate the sequence if it is too long."""
sequence = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
expected_output = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
self.assertEqual(truncate_or_pad(sequence, self.block_size, 0), expected_output)
diff --git a/examples/research_projects/bertabs/utils_summarization.py b/examples/research_projects/bertabs/utils_summarization.py
index 11ce5994297d..716365336bb5 100644
--- a/examples/research_projects/bertabs/utils_summarization.py
+++ b/examples/research_projects/bertabs/utils_summarization.py
@@ -47,7 +47,7 @@ def __init__(self, path="", prefix="train"):
self.documents.append(path_to_story)
def __len__(self):
- """ Returns the number of documents. """
+ """Returns the number of documents."""
return len(self.documents)
def __getitem__(self, idx):
diff --git a/examples/research_projects/bertology/run_bertology.py b/examples/research_projects/bertology/run_bertology.py
index d0eef3043007..fb1c24e5bc6e 100644
--- a/examples/research_projects/bertology/run_bertology.py
+++ b/examples/research_projects/bertology/run_bertology.py
@@ -49,14 +49,14 @@
def entropy(p):
- """ Compute the entropy of a probability distribution """
+ """Compute the entropy of a probability distribution"""
plogp = p * torch.log(p)
plogp[p == 0] = 0
return -plogp.sum(dim=-1)
def print_2d_tensor(tensor):
- """ Print a 2D tensor """
+ """Print a 2D tensor"""
logger.info("lv, h >\t" + "\t".join(f"{x + 1}" for x in range(len(tensor))))
for row in range(len(tensor)):
if tensor.dtype != torch.long:
diff --git a/examples/research_projects/bertology/run_prune_gpt.py b/examples/research_projects/bertology/run_prune_gpt.py
index 7e88f3081e95..5dbabe39128f 100644
--- a/examples/research_projects/bertology/run_prune_gpt.py
+++ b/examples/research_projects/bertology/run_prune_gpt.py
@@ -36,7 +36,7 @@ def save_model(model, dirpath):
def entropy(p, unlogit=False):
- """ Compute the entropy of a probability distribution """
+ """Compute the entropy of a probability distribution"""
exponent = 2
if unlogit:
p = torch.pow(p, exponent)
@@ -46,7 +46,7 @@ def entropy(p, unlogit=False):
def print_2d_tensor(tensor):
- """ Print a 2D tensor """
+ """Print a 2D tensor"""
logger.info("lv, h >\t" + "\t".join(f"{x + 1}" for x in range(len(tensor))))
for row in range(len(tensor)):
if tensor.dtype != torch.long:
diff --git a/examples/research_projects/deebert/run_glue_deebert.py b/examples/research_projects/deebert/run_glue_deebert.py
index 7e415d093969..97ae17faab24 100644
--- a/examples/research_projects/deebert/run_glue_deebert.py
+++ b/examples/research_projects/deebert/run_glue_deebert.py
@@ -70,7 +70,7 @@ def get_wanted_result(result):
def train(args, train_dataset, model, tokenizer, train_highway=False):
- """ Train the model """
+ """Train the model"""
if args.local_rank in [-1, 0]:
tb_writer = SummaryWriter()
diff --git a/examples/research_projects/distillation/README.md b/examples/research_projects/distillation/README.md
index 3dc2c53a1e31..36b45f79889f 100644
--- a/examples/research_projects/distillation/README.md
+++ b/examples/research_projects/distillation/README.md
@@ -163,7 +163,7 @@ python -m torch.distributed.launch \
--master_port $MASTER_PORT \
train.py \
--force \
- --gpus $WORLD_SIZE \
+ --n_gpu $WORLD_SIZE \
--student_type distilbert \
--student_config training_configs/distilbert-base-uncased.json \
--teacher_type bert \
diff --git a/examples/research_projects/distillation/run_squad_w_distillation.py b/examples/research_projects/distillation/run_squad_w_distillation.py
index 3429bf1cbe79..1c7256fccfed 100644
--- a/examples/research_projects/distillation/run_squad_w_distillation.py
+++ b/examples/research_projects/distillation/run_squad_w_distillation.py
@@ -92,7 +92,7 @@ def to_list(tensor):
def train(args, train_dataset, model, tokenizer, teacher=None):
- """ Train the model """
+ """Train the model"""
if args.local_rank in [-1, 0]:
tb_writer = SummaryWriter()
diff --git a/examples/research_projects/distillation/train.py b/examples/research_projects/distillation/train.py
index ce5df3319849..0d21ae04f8af 100644
--- a/examples/research_projects/distillation/train.py
+++ b/examples/research_projects/distillation/train.py
@@ -210,7 +210,7 @@ def main():
help="For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']."
"See details at https://nvidia.github.io/apex/amp.html",
)
- parser.add_argument("--gpus", type=int, default=1, help="Number of GPUs in the node.")
+ parser.add_argument("--n_gpu", type=int, default=1, help="Number of GPUs in the node.")
parser.add_argument("--local_rank", type=int, default=-1, help="Distributed training - Local rank")
parser.add_argument("--seed", type=int, default=56, help="Random seed")
diff --git a/examples/research_projects/mm-imdb/run_mmimdb.py b/examples/research_projects/mm-imdb/run_mmimdb.py
index d948a5a62dd9..4157d2e9cfb8 100644
--- a/examples/research_projects/mm-imdb/run_mmimdb.py
+++ b/examples/research_projects/mm-imdb/run_mmimdb.py
@@ -64,7 +64,7 @@ def set_seed(args):
def train(args, train_dataset, model, tokenizer, criterion):
- """ Train the model """
+ """Train the model"""
if args.local_rank in [-1, 0]:
tb_writer = SummaryWriter()
diff --git a/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py b/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py
index c686d39e344f..0f4803cdd5c0 100644
--- a/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py
+++ b/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py
@@ -393,7 +393,7 @@ class MaskedBertPreTrainedModel(PreTrainedModel):
base_model_prefix = "bert"
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, (nn.Linear, nn.Embedding)):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/examples/research_projects/movement-pruning/masked_run_glue.py b/examples/research_projects/movement-pruning/masked_run_glue.py
index 0657aa24ce35..48605ee05316 100644
--- a/examples/research_projects/movement-pruning/masked_run_glue.py
+++ b/examples/research_projects/movement-pruning/masked_run_glue.py
@@ -105,7 +105,7 @@ def regularization(model: nn.Module, mode: str):
def train(args, train_dataset, model, tokenizer, teacher=None):
- """ Train the model """
+ """Train the model"""
if args.local_rank in [-1, 0]:
tb_writer = SummaryWriter(log_dir=args.output_dir)
diff --git a/examples/research_projects/movement-pruning/masked_run_squad.py b/examples/research_projects/movement-pruning/masked_run_squad.py
index 9fd219c08906..56f26eff1051 100644
--- a/examples/research_projects/movement-pruning/masked_run_squad.py
+++ b/examples/research_projects/movement-pruning/masked_run_squad.py
@@ -113,7 +113,7 @@ def to_list(tensor):
def train(args, train_dataset, model, tokenizer, teacher=None):
- """ Train the model """
+ """Train the model"""
if args.local_rank in [-1, 0]:
tb_writer = SummaryWriter(log_dir=args.output_dir)
diff --git a/examples/tensorflow/text-classification/run_text_classification.py b/examples/tensorflow/text-classification/run_text_classification.py
index 3c9e2600970d..f725f1c930f0 100644
--- a/examples/tensorflow/text-classification/run_text_classification.py
+++ b/examples/tensorflow/text-classification/run_text_classification.py
@@ -164,17 +164,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of predict examples to this "
"value if set."
},
)
@@ -468,13 +468,13 @@ def preprocess_function(examples):
if "validation" in datasets:
eval_dataset = datasets["validation"]
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
if "test" in datasets:
- test_dataset = datasets["test"]
- if data_args.max_test_samples is not None:
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
+ predict_dataset = datasets["test"]
+ if data_args.max_predict_samples is not None:
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
# endregion
@@ -513,15 +513,15 @@ def preprocess_function(examples):
# region Prediction
if "test" in datasets:
- logger.info("Doing predictions on test dataset...")
+ logger.info("Doing predictions on Predict dataset...")
- test_dataset = DataSequence(
- test_dataset, non_label_column_names, batch_size=training_args.per_device_eval_batch_size, labels=False
+ predict_dataset = DataSequence(
+ predict_dataset, non_label_column_names, batch_size=training_args.per_device_eval_batch_size, labels=False
)
- predictions = model.predict(test_dataset)["logits"]
+ predictions = model.predict(predict_dataset)["logits"]
predictions = np.squeeze(predictions) if is_regression else np.argmax(predictions, axis=1)
- output_test_file = os.path.join(training_args.output_dir, "test_results.txt")
- with open(output_test_file, "w") as writer:
+ output_predict_file = os.path.join(training_args.output_dir, "predict_results.txt")
+ with open(output_predict_file, "w") as writer:
writer.write("index\tprediction\n")
for index, item in enumerate(predictions):
if is_regression:
@@ -529,7 +529,7 @@ def preprocess_function(examples):
else:
item = model.config.id2label[item]
writer.write(f"{index}\t{item}\n")
- logger.info(f"Wrote predictions to {output_test_file}!")
+ logger.info(f"Wrote predictions to {output_predict_file}!")
# endregion
diff --git a/setup.py b/setup.py
index cebcb65d096b..b4d65585b83a 100644
--- a/setup.py
+++ b/setup.py
@@ -86,11 +86,11 @@
# 2. once modified, run: `make deps_table_update` to update src/transformers/dependency_versions_table.py
_deps = [
"Pillow",
- "black>=20.8b1",
+ "black==21.4b0",
"cookiecutter==1.7.2",
"dataclasses",
"datasets",
- "deepspeed>=0.3.14",
+ "deepspeed>=0.3.15",
"docutils==0.16.0",
"fairscale>0.3",
"faiss-cpu",
diff --git a/src/transformers/commands/add_new_model.py b/src/transformers/commands/add_new_model.py
index 66dc7201e6cd..9cac3df69ca2 100644
--- a/src/transformers/commands/add_new_model.py
+++ b/src/transformers/commands/add_new_model.py
@@ -57,14 +57,14 @@ def run(self):
if not _has_cookiecutter:
raise ImportError(
"Model creation dependencies are required to use the `add_new_model` command. Install them by running "
- "the folowing at the root of your `transformers` clone:\n\n\t$ pip install -e .[modelcreation]\n"
+ "the following at the root of your `transformers` clone:\n\n\t$ pip install -e .[modelcreation]\n"
)
# Ensure that there is no other `cookiecutter-template-xxx` directory in the current working directory
directories = [directory for directory in os.listdir() if "cookiecutter-template-" == directory[:22]]
if len(directories) > 0:
raise ValueError(
"Several directories starting with `cookiecutter-template-` in current working directory. "
- "Please clean your directory by removing all folders startign with `cookiecutter-template-` or "
+ "Please clean your directory by removing all folders starting with `cookiecutter-template-` or "
"change your working directory."
)
diff --git a/src/transformers/commands/lfs.py b/src/transformers/commands/lfs.py
index 42b00f0d2fae..9d8f90502f4f 100644
--- a/src/transformers/commands/lfs.py
+++ b/src/transformers/commands/lfs.py
@@ -96,7 +96,7 @@ def write_msg(msg: Dict):
def read_msg() -> Optional[Dict]:
- """Read Line delimited JSON from stdin. """
+ """Read Line delimited JSON from stdin."""
msg = json.loads(sys.stdin.readline().strip())
if "terminate" in (msg.get("type"), msg.get("event")):
diff --git a/src/transformers/data/processors/squad.py b/src/transformers/data/processors/squad.py
index 54134bfa45ac..cea84fb3b11a 100644
--- a/src/transformers/data/processors/squad.py
+++ b/src/transformers/data/processors/squad.py
@@ -244,7 +244,7 @@ def squad_convert_example_to_features(
cls_index = span["input_ids"].index(tokenizer.cls_token_id)
# p_mask: mask with 1 for token than cannot be in the answer (0 for token which can be in an answer)
- # Original TF implem also keep the classification token (set to 0)
+ # Original TF implementation also keep the classification token (set to 0)
p_mask = np.ones_like(span["token_type_ids"])
if tokenizer.padding_side == "right":
p_mask[len(truncated_query) + sequence_added_tokens :] = 0
diff --git a/src/transformers/data/processors/utils.py b/src/transformers/data/processors/utils.py
index 06db91f7e27b..a5a04266a014 100644
--- a/src/transformers/data/processors/utils.py
+++ b/src/transformers/data/processors/utils.py
@@ -124,7 +124,7 @@ def _read_tsv(cls, input_file, quotechar=None):
class SingleSentenceClassificationProcessor(DataProcessor):
- """ Generic processor for a single sentence classification data set."""
+ """Generic processor for a single sentence classification data set."""
def __init__(self, labels=None, examples=None, mode="classification", verbose=False):
self.labels = [] if labels is None else labels
diff --git a/src/transformers/dependency_versions_table.py b/src/transformers/dependency_versions_table.py
index 3b7d158d1e23..02c302755a5d 100644
--- a/src/transformers/dependency_versions_table.py
+++ b/src/transformers/dependency_versions_table.py
@@ -3,11 +3,11 @@
# 2. run `make deps_table_update``
deps = {
"Pillow": "Pillow",
- "black": "black>=20.8b1",
+ "black": "black==21.4b0",
"cookiecutter": "cookiecutter==1.7.2",
"dataclasses": "dataclasses",
"datasets": "datasets",
- "deepspeed": "deepspeed>=0.3.14",
+ "deepspeed": "deepspeed>=0.3.15",
"docutils": "docutils==0.16.0",
"fairscale": "fairscale>0.3",
"faiss-cpu": "faiss-cpu",
diff --git a/src/transformers/feature_extraction_sequence_utils.py b/src/transformers/feature_extraction_sequence_utils.py
index ec99f152f457..2d0312124068 100644
--- a/src/transformers/feature_extraction_sequence_utils.py
+++ b/src/transformers/feature_extraction_sequence_utils.py
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""
- Sequence feature extraction class for common feature extrcactors to preprocess sequences.
+ Sequence feature extraction class for common feature extractors to preprocess sequences.
"""
from typing import Dict, List, Optional, Union
diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py
index 392728fdf0e9..a16ace202ce2 100644
--- a/src/transformers/file_utils.py
+++ b/src/transformers/file_utils.py
@@ -551,7 +551,7 @@ def wrapper(*args, **kwargs):
("sklearn", (is_sklearn_available, SKLEARN_IMPORT_ERROR)),
("speech", (is_speech_available, SPEECH_IMPORT_ERROR)),
("tf", (is_tf_available, TENSORFLOW_IMPORT_ERROR)),
- ("tokenziers", (is_tokenizers_available, TOKENIZERS_IMPORT_ERROR)),
+ ("tokenizers", (is_tokenizers_available, TOKENIZERS_IMPORT_ERROR)),
("torch", (is_torch_available, PYTORCH_IMPORT_ERROR)),
("vision", (is_vision_available, VISION_IMPORT_ERROR)),
]
@@ -1462,7 +1462,7 @@ def wrapper(*args, **kwargs):
def is_tensor(x):
- """ Tests if ``x`` is a :obj:`torch.Tensor`, :obj:`tf.Tensor` or :obj:`np.ndarray`. """
+ """Tests if ``x`` is a :obj:`torch.Tensor`, :obj:`tf.Tensor` or :obj:`np.ndarray`."""
if is_torch_available():
import torch
@@ -1683,8 +1683,19 @@ def _get_module(self, module_name: str) -> ModuleType:
raise NotImplementedError
+class AggregationStrategy(ExplicitEnum):
+ """
+ Possible values for the ``aggregation_strategy`` argument in :meth:`TokenClassificationPipeline.__init__`. Useful
+ for tab-completion in an IDE.
+ """
+
+ FIRST = "first"
+ MAX = "max"
+ AVERAGE = "average"
+
+
def copy_func(f):
- """ Returns a copy of a function f."""
+ """Returns a copy of a function f."""
# Based on http://stackoverflow.com/a/6528148/190597 (Glenn Maynard)
g = types.FunctionType(f.__code__, f.__globals__, name=f.__name__, argdefs=f.__defaults__, closure=f.__closure__)
g = functools.update_wrapper(g, f)
diff --git a/src/transformers/generation_beam_search.py b/src/transformers/generation_beam_search.py
index 1fea43e1d7e5..cebe754af23e 100644
--- a/src/transformers/generation_beam_search.py
+++ b/src/transformers/generation_beam_search.py
@@ -13,6 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+import warnings
from abc import ABC, abstractmethod
from collections import UserDict
from typing import Optional, Tuple
@@ -110,6 +111,7 @@ def finalize(
next_scores: torch.FloatTensor,
next_tokens: torch.LongTensor,
next_indices: torch.LongTensor,
+ max_length: int,
**kwargs
) -> torch.LongTensor:
raise NotImplementedError("This is an abstract method.")
@@ -152,15 +154,14 @@ class BeamSearchScorer(BeamScorer):
def __init__(
self,
batch_size: int,
- max_length: int,
num_beams: int,
device: torch.device,
length_penalty: Optional[float] = 1.0,
do_early_stopping: Optional[bool] = False,
num_beam_hyps_to_keep: Optional[int] = 1,
num_beam_groups: Optional[int] = 1,
+ **kwargs,
):
- self.max_length = max_length
self.num_beams = num_beams
self.device = device
self.length_penalty = length_penalty
@@ -173,7 +174,6 @@ def __init__(
self._beam_hyps = [
BeamHypotheses(
num_beams=self.num_beams,
- max_length=self.max_length,
length_penalty=self.length_penalty,
early_stopping=self.do_early_stopping,
)
@@ -192,6 +192,13 @@ def __init__(
f"has to be divisible by `num_beam_groups`, but is {num_beam_groups} with `num_beams` being {num_beams}."
)
+ if "max_length" in kwargs:
+ warnings.warn(
+ "Passing `max_length` to BeamSearchScorer is deprecated and has no effect."
+ "`max_length` should be passed directly to `beam_search(...)`, `beam_sample(...)`"
+ ",or `group_beam_search(...)`."
+ )
+
@property
def is_done(self) -> bool:
return self._done.all()
@@ -279,6 +286,7 @@ def finalize(
final_beam_scores: torch.FloatTensor,
final_beam_tokens: torch.LongTensor,
final_beam_indices: torch.LongTensor,
+ max_length: int,
pad_token_id: Optional[int] = None,
eos_token_id: Optional[int] = None,
) -> Tuple[torch.LongTensor]:
@@ -316,7 +324,7 @@ def finalize(
best_scores[i * self.num_beam_hyps_to_keep + j] = best_score
# prepare for adding eos
- sent_max_len = min(sent_lengths.max().item() + 1, self.max_length)
+ sent_max_len = min(sent_lengths.max().item() + 1, max_length)
decoded: torch.LongTensor = input_ids.new(batch_size * self.num_beam_hyps_to_keep, sent_max_len)
# shorter batches are padded if needed
if sent_lengths.min().item() != sent_lengths.max().item():
@@ -326,7 +334,7 @@ def finalize(
# fill with hypotheses and eos_token_id if the latter fits in
for i, hypo in enumerate(best):
decoded[i, : sent_lengths[i]] = hypo
- if sent_lengths[i] < self.max_length:
+ if sent_lengths[i] < max_length:
decoded[i, sent_lengths[i]] = eos_token_id
return UserDict(
{
@@ -337,11 +345,10 @@ def finalize(
class BeamHypotheses:
- def __init__(self, num_beams: int, max_length: int, length_penalty: float, early_stopping: bool):
+ def __init__(self, num_beams: int, length_penalty: float, early_stopping: bool):
"""
Initialize n-best list of hypotheses.
"""
- self.max_length = max_length - 1 # ignoring bos_token
self.length_penalty = length_penalty
self.early_stopping = early_stopping
self.num_beams = num_beams
diff --git a/src/transformers/generation_logits_process.py b/src/transformers/generation_logits_process.py
index 1d790b287823..1b98909955fc 100644
--- a/src/transformers/generation_logits_process.py
+++ b/src/transformers/generation_logits_process.py
@@ -446,7 +446,7 @@ def _set_scores_to_inf_for_banned_tokens(self, scores: torch.Tensor, banned_toke
class PrefixConstrainedLogitsProcessor(LogitsProcessor):
r"""
- :class:`transformers.LogitsProcessor` that enforces contrained generation and is useful for prefix-conditioned
+ :class:`transformers.LogitsProcessor` that enforces constrained generation and is useful for prefix-conditioned
constrained generation. See `Autoregressive Entity Retrieval `__ for more
information.
diff --git a/src/transformers/generation_stopping_criteria.py b/src/transformers/generation_stopping_criteria.py
index ab853985240d..65fef72464ee 100644
--- a/src/transformers/generation_stopping_criteria.py
+++ b/src/transformers/generation_stopping_criteria.py
@@ -23,7 +23,7 @@
Prediction scores of a language modeling head. These can be scores for each vocabulary token before SoftMax
or scores for each vocabulary token after SoftMax.
kwargs:
- Additional stopping critera specific kwargs.
+ Additional stopping criteria specific kwargs.
Return:
:obj:`bool`. :obj:`False` indicates we should continue, :obj:`True` indicates we should stop.
diff --git a/src/transformers/generation_tf_utils.py b/src/transformers/generation_tf_utils.py
index 8c0802e95254..7469521b3996 100644
--- a/src/transformers/generation_tf_utils.py
+++ b/src/transformers/generation_tf_utils.py
@@ -442,8 +442,8 @@ def _generate_no_beam_search(
**kwargs
):
"""
- Generate sequences for each example without beam search (num_beams == 1). All returned sequence are generated
- independantly.
+ Generate sequences for each example without beam search (num_beams == 1). All returned sequences are generated
+ independently.
"""
# length of generated sentences / unfinished sentences
diff --git a/src/transformers/generation_utils.py b/src/transformers/generation_utils.py
index 959c63c72847..9f21ee104a60 100644
--- a/src/transformers/generation_utils.py
+++ b/src/transformers/generation_utils.py
@@ -821,7 +821,7 @@ def generate(
... "at least two people were killed in a suspected bomb attack on a passenger bus "
... "in the strife-torn southern philippines on monday , the military said."
... )
- >>> # encode input contex
+ >>> # encode input context
>>> input_ids = tokenizer(document, return_tensors="pt").input_ids
>>> # generate 3 independent sequences using beam search decoding (5 beams)
>>> # with T5 encoder-decoder model conditioned on short news article.
@@ -1027,7 +1027,6 @@ def generate(
beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=stopping_criteria.max_length,
num_beams=num_beams,
device=self.device,
length_penalty=length_penalty,
@@ -1063,7 +1062,6 @@ def generate(
raise ValueError("`max_length` needs to be a stopping_criteria for now.")
beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=stopping_criteria.max_length,
num_beams=num_beams,
device=self.device,
length_penalty=length_penalty,
@@ -1700,7 +1698,6 @@ def beam_search(
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
... batch_size=1,
- ... max_length=model.config.max_length,
... num_beams=num_beams,
... device=model.device,
... )
@@ -1756,7 +1753,7 @@ def beam_search(
assert (
num_beams * batch_size == batch_beam_size
- ), "Batch dimension of `input_ids` should be {num_beams * batch_size}, but is {batch_beam_size}."
+ ), f"Batch dimension of `input_ids` should be {num_beams * batch_size}, but is {batch_beam_size}."
beam_scores = torch.zeros((batch_size, num_beams), dtype=torch.float, device=input_ids.device)
beam_scores[:, 1:] = -1e9
@@ -1792,10 +1789,7 @@ def beam_search(
# hack: adjust tokens for Marian. For Marian we have to make sure that the `pad_token_id`
# cannot be generated both before and after the `F.log_softmax` operation.
- next_token_logits = self.adjust_logits_during_generation(
- next_token_logits, cur_len=cur_len, max_length=None
- )
-
+ next_token_logits = self.adjust_logits_during_generation(next_token_logits, cur_len=cur_len)
next_token_scores = F.log_softmax(next_token_logits, dim=-1) # (batch_size * num_beams, vocab_size)
next_token_scores = logits_processor(input_ids, next_token_scores)
@@ -1861,7 +1855,13 @@ def beam_search(
this_peer_finished = True
sequence_outputs = beam_scorer.finalize(
- input_ids, beam_scores, next_tokens, next_indices, pad_token_id=pad_token_id, eos_token_id=eos_token_id
+ input_ids,
+ beam_scores,
+ next_tokens,
+ next_indices,
+ pad_token_id=pad_token_id,
+ eos_token_id=eos_token_id,
+ max_length=stopping_criteria.max_length,
)
if return_dict_in_generate:
@@ -2086,10 +2086,7 @@ def beam_sample(
# hack: adjust tokens for Marian. For Marian we have to make sure that the `pad_token_id`
# cannot be generated both before and after the `F.log_softmax` operation.
- next_token_logits = self.adjust_logits_during_generation(
- next_token_logits, cur_len=cur_len, max_length=None
- )
-
+ next_token_logits = self.adjust_logits_during_generation(next_token_logits, cur_len=cur_len)
next_token_scores = F.log_softmax(next_token_logits, dim=-1) # (batch_size * num_beams, vocab_size)
next_token_scores = logits_processor(input_ids, next_token_scores)
@@ -2160,7 +2157,13 @@ def beam_sample(
this_peer_finished = True
sequence_outputs = beam_scorer.finalize(
- input_ids, beam_scores, next_tokens, next_indices, pad_token_id=pad_token_id, eos_token_id=eos_token_id
+ input_ids,
+ beam_scores,
+ next_tokens,
+ next_indices,
+ pad_token_id=pad_token_id,
+ eos_token_id=eos_token_id,
+ max_length=stopping_criteria.max_length,
)
if return_dict_in_generate:
@@ -2411,10 +2414,7 @@ def group_beam_search(
# hack: adjust tokens for Marian. For Marian we have to make sure that the `pad_token_id`
# cannot be generated both before and after the `F.log_softmax` operation.
- next_token_logits = self.adjust_logits_during_generation(
- next_token_logits, cur_len=cur_len, max_length=None
- )
-
+ next_token_logits = self.adjust_logits_during_generation(next_token_logits, cur_len=cur_len)
next_token_scores = F.log_softmax(next_token_logits, dim=-1) # (batch_size * group_size, vocab_size)
vocab_size = next_token_scores.shape[-1]
@@ -2497,7 +2497,13 @@ def group_beam_search(
this_peer_finished = True
sequence_outputs = beam_scorer.finalize(
- input_ids, beam_scores, next_tokens, next_indices, pad_token_id=pad_token_id, eos_token_id=eos_token_id
+ input_ids,
+ beam_scores,
+ next_tokens,
+ next_indices,
+ pad_token_id=pad_token_id,
+ eos_token_id=eos_token_id,
+ max_length=stopping_criteria.max_length,
)
if return_dict_in_generate:
diff --git a/src/transformers/integrations.py b/src/transformers/integrations.py
index 7e4ab0f5c7a1..a2d6743a1e2a 100644
--- a/src/transformers/integrations.py
+++ b/src/transformers/integrations.py
@@ -19,8 +19,8 @@
import json
import numbers
import os
-import sys
import tempfile
+import weakref
from copy import deepcopy
from pathlib import Path
@@ -269,74 +269,180 @@ def rewrite_logs(d):
return new_d
-_is_deepspeed_zero3_enabled = None
+def _is_true(config, key):
+ if config is None:
+ return False
+ return bool(config.get(key))
-def is_deepspeed_zero3_enabled():
- """
- This function answers to the question of whether DeepSpeed is going to be used and run using ZeRO Stage 3.
+def _set_if_auto(config, key, val):
+ if config is None:
+ return
+ if config.get(key) == "auto":
+ config[key] = val
- It includes an auto-discovery method, see comments in the code for details.
- Returns: ``True`` if either it was explicitly enabled via ``deepspeed_zero3_enable(True)`` or the auto-detector was
- able to derive that the ``Trainer`` will be running via DeepSpeed ZeRO stage 3.
+class DeepSpeedConfigHF:
"""
- global _is_deepspeed_zero3_enabled
- if _is_deepspeed_zero3_enabled is None:
- _is_deepspeed_zero3_enabled = False
- # Try to auto-discover if we are about to use DeepSpeed with ZeRO3 enabled. This will only
- # work for scripts using cli to pass --deepspeed ds_config.json. If cmd args aren't used,
- # then to get the model efficiently loaded across multiple-gpus one has to explicitly call
- # is_deepspeed_zero3_enabled(True) **before** instantiating a model object
- if "--deepspeed" in sys.argv:
- idx = sys.argv.index("--deepspeed")
- ds_config = sys.argv[idx + 1]
- if not os.path.exists(ds_config):
- raise ValueError("--deepspeed requires a valid path to a config file")
- config = deepspeed_parse_config(ds_config)
- if (
- "zero_optimization" in config
- and "stage" in config["zero_optimization"]
- and config["zero_optimization"]["stage"] == 3
- ):
- _is_deepspeed_zero3_enabled = True
-
- return _is_deepspeed_zero3_enabled
-
-
-def deepspeed_zero3_enable(enable=True):
- """
- ``is_deepspeed_zero3_enabled()`` tries to derive automatically if DeepSpeed ZeRO 3 is going to be used by looking
- at ``sys.argv`` which may or may contain information about where to find the DeepSpeed config if any.
+ This object contains Deepspeed configuration and can be quickly queried for things like zero stage.
- This function allows for explicit enabling/disabling of this global flag.
+ We store a ``weakref`` of this object in the module's global to be able to access the config from areas where the
+ Trainer is not available (e.g. `from_pretrained` and `_get_resized_embeddings`).
- Args:
- enable: if set to ``True`` will make ``is_deepspeed_zero3_enabled()`` return ``True``
+ The ``DeepSpeedConfigHF`` object is meant to be created during ``TrainingArguments`` object creation and has the
+ same lifespan as the latter.
"""
- global _is_deepspeed_zero3_enabled
- _is_deepspeed_zero3_enabled = enable
+ def __init__(self, args):
+ self.config = None
+ self.stage = 0
+ self.offload = False
-def deepspeed_parse_config(ds_config):
- """
- If ``ds_config`` isn't already a dict, read it from the config file.
+ dep_version_check("deepspeed")
- If it's already a dict, return a copy of it, so that we can freely modify it.
- """
- dep_version_check("deepspeed")
-
- if isinstance(ds_config, dict):
- # Don't modify user's data should they want to reuse it (e.g. in tests), because once we
- # modified it, it will not be accepted here again, since some config params must be not set by users
- config = deepcopy(ds_config)
- elif isinstance(ds_config, str):
- with io.open(ds_config, "r", encoding="utf-8") as f:
- config = json.load(f)
+ self.config_process(args)
+
+ # set global weakref object
+ deepspeed_config_hf_set(self)
+
+ def is_zero2(self):
+ return self.stage == 2
+
+ def is_zero3(self):
+ return self.stage == 3
+
+ def is_offload(self):
+ return self.offload
+
+ def config_process(self, args):
+ """
+ 1. load json if the ``args.deepspeed`` is a path
+ 2. replace any ``auto`` values in the config with the correct or recommended value
+
+ This is done as early as possible, before model is created, to allow ``is_deepspeed_zero3_enabled`` query and
+ getting to the early deepspeed config object during ``zero.Init()`` which needs whether fp16 is enabled, dtype,
+ etc.
+
+ """
+ config_file_or_dict = args.deepspeed
+ if isinstance(config_file_or_dict, dict):
+ # Don't modify user's data should they want to reuse it (e.g. in tests), because once we
+ # modified it, it will not be accepted here again, since `auto` values would have been overriden
+ config = deepcopy(config_file_or_dict)
+ elif isinstance(config_file_or_dict, str):
+ with io.open(config_file_or_dict, "r", encoding="utf-8") as f:
+ config = json.load(f)
+ else:
+ raise ValueError("expecting either a path to a config file or a pre-populated dict")
+
+ self.config = config
+
+ # DeepSpeed does:
+ # train_batch_size = world_size * train_micro_batch_size_per_gpu * gradient_accumulation_steps
+ train_batch_size = args.world_size * args.per_device_train_batch_size * args.gradient_accumulation_steps
+ _set_if_auto(config, "train_micro_batch_size_per_gpu", args.per_device_train_batch_size)
+ _set_if_auto(config, "gradient_accumulation_steps", args.gradient_accumulation_steps)
+ _set_if_auto(config, "train_batch_size", train_batch_size)
+ _set_if_auto(config, "gradient_clipping", args.max_grad_norm)
+
+ # zero
+ config_zero = config.get("zero_optimization", {})
+ self.stage = config_zero.get("stage", 0)
+
+ config_optim = config.get("optimizer", {})
+ if config_optim != {}:
+ config_optim_params = config_optim.get("params")
+ _set_if_auto(config_optim_params, "lr", args.learning_rate)
+ _set_if_auto(config_optim_params, "betas", [args.adam_beta1, args.adam_beta2])
+ _set_if_auto(config_optim_params, "eps", args.adam_epsilon)
+ _set_if_auto(config_optim_params, "weight_decay", args.weight_decay)
+
+ config_sched = config.get("scheduler", {})
+ if config_sched != {}:
+ config_sched_params = config_sched.get("params")
+ _set_if_auto(config_sched_params, "warmup_min_lr", 0)
+ _set_if_auto(config_sched_params, "warmup_max_lr", args.learning_rate)
+ _set_if_auto(config_sched_params, "warmup_num_steps", args.warmup_steps)
+ # total_num_steps - will get set in deepspeed_init
+
+ # fp16
+ if args.fp16:
+ fp16_backend = "apex" if args.fp16_backend == "apex" else "amp"
+ else:
+ fp16_backend = None
+
+ # amp: similar to the pytorch native amp - it has a bunch of optional params but we won't set
+ # any here unless the user did the work
+ config_fp16 = config.get("fp16")
+ # XXX: at the moment fp16 can't be False, but the fp32 solution is in works - once it's PR'ed and
+ # merged and a new release is made, delete the next line and uncomment the one after it
+ _set_if_auto(config_fp16, "enabled", True)
+ # _set_if_auto(config_fp16, "enabled", fp16_backend == "amp")
+
+ # apex: delegates amp work to apex (which needs to be available), but it cannot be used with any
+ # ZeRO features, so probably best to be avoided.
+ config_amp = config.get("amp")
+ _set_if_auto(config_amp, "enabled", fp16_backend == "apex")
+ _set_if_auto(config_amp, "opt_level", args.fp16_opt_level)
+
+ config_zero = config.get("zero_optimization", {})
+ if self.is_zero2():
+ self.offload = _is_true(config_zero, "cpu_offload")
+ elif self.is_zero3():
+ offload_devices = ["cpu", "nvme"]
+ if config_zero.get("offload_optimizer", {}).get("device") in offload_devices:
+ self.offload = True
+ if config_zero.get("offload_param", {}).get("device") in offload_devices:
+ self.offload = True
+
+ def config_finalize(self, args, model, num_training_steps):
+ """
+ This stage is run after we have the model and know num_training_steps.
+
+ Now we we can complete the configuration process.
+
+ """
+ config = self.config
+
+ # zero
+ config_zero = config.get("zero_optimization", {})
+ if self.is_zero3():
+ # automatically assign the optimal config values based on model config
+ hidden_size = model.config.hidden_size
+ _set_if_auto(config_zero, "reduce_bucket_size", hidden_size * hidden_size)
+ _set_if_auto(config_zero, "stage3_prefetch_bucket_size", 0.9 * hidden_size * hidden_size)
+ _set_if_auto(config_zero, "stage3_param_persistence_threshold", 10 * hidden_size)
+
+ # scheduler
+ config_sched = config.get("scheduler", {})
+ config_sched_params = config_sched.get("params", {})
+ _set_if_auto(config_sched_params, "total_num_steps", num_training_steps)
+
+
+# keep the config object global to be able to access it anywhere during TrainingArguments life-cycle
+_deepspeed_config_hf_weak_ref = None
+
+
+def deepspeed_config_hf_set(deepspeed_config_hf_obj):
+ # this is a special weakref global object to allow us to get to Deepspeed config from APIs
+ # that don't have an easy way to get to the Deepspeed config outside of the Trainer domain.
+ global _deepspeed_config_hf_weak_ref
+ # will go away automatically when DeepSpeedConfigHF is destroyed (when TrainingArguments is destroyed)
+ _deepspeed_config_hf_weak_ref = weakref.ref(deepspeed_config_hf_obj)
+
+
+def is_deepspeed_zero3_enabled():
+ if _deepspeed_config_hf_weak_ref is not None and _deepspeed_config_hf_weak_ref() is not None:
+ return _deepspeed_config_hf_weak_ref().is_zero3()
else:
- raise ValueError("expecting either a path to a config file or a pre-populated dict")
+ return False
+
- return config
+def deepspeed_config():
+ if _deepspeed_config_hf_weak_ref is not None and _deepspeed_config_hf_weak_ref() is not None:
+ return _deepspeed_config_hf_weak_ref().config
+ else:
+ return None
def deepspeed_init(trainer, num_training_steps, resume_from_checkpoint=None):
@@ -355,41 +461,16 @@ def deepspeed_init(trainer, num_training_steps, resume_from_checkpoint=None):
"""
import deepspeed
- args = trainer.args
model = trainer.model
- config = deepspeed_parse_config(args.deepspeed)
+ deepspeed_config_hf = trainer.args.deepspeed_config_hf
+ deepspeed_config_hf.config_finalize(trainer.args, model, num_training_steps)
- # The following code translates relevant trainer's cl args into the DS config
-
- # First to ensure that there is no mismatch between cl args values and presets in the config
- # file, ask to not set in ds config file:
- # - "train_batch_size",
- # - "train_micro_batch_size_per_gpu",
- # - "gradient_accumulation_steps"
- bs_keys = ["train_batch_size", "train_micro_batch_size_per_gpu"]
- if len([x for x in bs_keys if x in config.keys()]):
- raise ValueError(
- f"Do not include {bs_keys} entries in the ds config file, as they will be set via --per_device_train_batch_size or its default"
- )
- if "gradient_accumulation_steps" in config.keys():
- raise ValueError(
- "Do not include gradient_accumulation_steps entries in the ds config file, as they will be set via --gradient_accumulation_steps or its default"
- )
-
- # DeepSpeed does:
- # train_batch_size = n_gpus * train_micro_batch_size_per_gpu * gradient_accumulation_steps
- # therefore we just need to set:
- config["train_micro_batch_size_per_gpu"] = args.per_device_train_batch_size
- config["gradient_accumulation_steps"] = args.gradient_accumulation_steps
-
- if "gradient_clipping" in config:
- logger.info("Keeping the `gradient_clipping` config intact, ignoring any gradient clipping-specific cl args")
- else: # override only if the ds config doesn't already have this section
- config["gradient_clipping"] = args.max_grad_norm
+ # resume config update - some bits like `model` and `num_training_steps` only become available during train
+ config = deepspeed_config_hf.config
# Optimizer + Scheduler
- # Currently support combos:
+ # Currently supported combos:
# 1. DS scheduler + DS optimizer: Yes
# 2. HF scheduler + HF optimizer: Yes
# 3. DS scheduler + HF optimizer: Yes
@@ -402,36 +483,16 @@ def deepspeed_init(trainer, num_training_steps, resume_from_checkpoint=None):
# 4. HF scheduler + DS optimizer: No
optimizer = None
- if "optimizer" in config:
- logger.info("Updating the `scheduler` config with other command line arguments")
-
- # to avoid inconsistent values of lr and warm up steps the command line args override config
- params = dict(
- lr=args.learning_rate,
- betas=[args.adam_beta1, args.adam_beta2],
- eps=args.adam_epsilon,
- weight_decay=args.weight_decay,
- )
- for k, v in params.items():
- if k in config["optimizer"]["params"]:
- logger.info(f"setting optimizer.params.{k} to {v}")
- config["optimizer"]["params"][k] = v
-
- else: # override only if the ds config doesn't already have this section
- if (
- "zero_optimization" in config
- and "cpu_offload" in config["zero_optimization"]
- and config["zero_optimization"]["cpu_offload"] is True
- ):
+ if "optimizer" not in config:
+ if deepspeed_config_hf.is_offload():
raise ValueError("ZeRO Offload can only work with DeepSpeed optimizers")
- else:
- # ds supports Adam, OneBitAdam, and Lamb optimizers and can import other optimizers from torch.
- # But trainer uses AdamW by default.
- # To use other optimizers so using a different scheduler requires voiding warranty with: `zero_allow_untested_optimizer`
- trainer.create_optimizer()
- optimizer = trainer.optimizer
- # flag that this is non-native optimizer
- config["zero_allow_untested_optimizer"] = True
+
+ # ds supports Adam, OneBitAdam, and Lamb optimizers and can import other optimizers from torch.
+ # But trainer uses AdamW by default.
+ trainer.create_optimizer()
+ optimizer = trainer.optimizer
+ # To use other optimizers requires voiding warranty with: `zero_allow_untested_optimizer`
+ config["zero_allow_untested_optimizer"] = True
# DS schedulers (deepspeed/runtime/lr_schedules.py):
#
@@ -442,25 +503,7 @@ def deepspeed_init(trainer, num_training_steps, resume_from_checkpoint=None):
# WarmupLR | constant_with_warmup | get_constant_schedule_with_warmup | w/ warmup_min_lr=0
# WarmupDecayLR| linear | get_linear_schedule_with_warmup |
lr_scheduler = None
- if "scheduler" in config:
- logger.info("Updating the `scheduler` config with other command line arguments")
- # the user won't easily know the correct num_training_steps should they use WarmupDecayLR,
- # so let's set it to the correct value
- if config["scheduler"]["type"] == "WarmupDecayLR":
- logger.info(f"setting scheduler.params.total_num_steps to {num_training_steps}")
- config["scheduler"]["params"]["total_num_steps"] = num_training_steps
-
- # to avoid inconsistent values of lr and warmup steps the command line args override config
- params = dict(
- warmup_max_lr=args.learning_rate,
- warmup_num_steps=args.warmup_steps,
- )
- for k, v in params.items():
- if k in config["scheduler"]["params"]:
- logger.info(f"setting scheduler.params.{k} to {v}")
- config["scheduler"]["params"][k] = v
-
- else: # override only if the ds config doesn't already have this section
+ if "scheduler" not in config:
if "optimizer" in config:
# to make this option work, we need to init DS optimizer first, then init HS scheduler,
# then pass the HS scheduler to DS init, which is not possible at the moment
@@ -469,43 +512,6 @@ def deepspeed_init(trainer, num_training_steps, resume_from_checkpoint=None):
trainer.create_scheduler(num_training_steps=num_training_steps)
lr_scheduler = trainer.lr_scheduler
- # fp16
- if trainer.fp16_backend is not None:
- # Deepspeed has 2 possible fp16 config entries:
- # - `fp16`: for the native amp - it has a bunch of optional params but we won't set any here unless the user did the work
- # - `amp`: which delegates amp work to apex (which needs to be available), but it cannot be used with any ZeRO features, so probably best to be avoided.
- if trainer.fp16_backend == "apex":
- if "amp" in config:
- logger.info("Keeping the `amp` config intact, ignoring any amp-specific cl args")
- else:
- config["amp"] = {
- "enabled": True,
- "opt_level": args.fp16_opt_level,
- }
- elif trainer.fp16_backend == "amp":
- if "fp16" in config:
- logger.info("Keeping the `fp16` config intact, ignoring any fp16-specific cl args")
- else:
- config["fp16"] = {
- "enabled": True,
- }
-
- # zero
- if "zero_optimization" in config:
- zero = config["zero_optimization"]
-
- # now we know for sure if zero3 is enabled
- deepspeed_zero3_enable(zero.get("stage") == 3)
-
- # automatically assign the optimal config values based on model config
- hidden_size = model.config.hidden_size
- if zero.get("reduce_bucket_size") == 0:
- zero["reduce_bucket_size"] = hidden_size * hidden_size
- if zero.get("stage3_prefetch_bucket_size") == 0:
- zero["stage3_prefetch_bucket_size"] = 0.9 * hidden_size * hidden_size
- if zero.get("stage3_param_persistence_threshold") == 0:
- zero["stage3_param_persistence_threshold"] = 10 * hidden_size
-
# keep for quick debug:
# from pprint import pprint; pprint(config)
@@ -856,8 +862,8 @@ def setup(self, args, state, model):
Whether to use MLflow .log_artifact() facility to log artifacts.
This only makes sense if logging to a remote server, e.g. s3 or GCS. If set to `True` or `1`, will copy
- whatever is in TrainerArgument's output_dir to the local or remote artifact storage. Using it without a
- remote storage will just copy the files to your artifact location.
+ whatever is in :class:`~transformers.TrainingArguments`'s ``output_dir`` to the local or remote
+ artifact storage. Using it without a remote storage will just copy the files to your artifact location.
"""
log_artifacts = os.getenv("HF_MLFLOW_LOG_ARTIFACTS", "FALSE").upper()
if log_artifacts in {"TRUE", "1"}:
diff --git a/src/transformers/modelcard.py b/src/transformers/modelcard.py
index 38316de8814a..97fdf1903ae6 100644
--- a/src/transformers/modelcard.py
+++ b/src/transformers/modelcard.py
@@ -215,6 +215,6 @@ def to_json_string(self):
return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
def to_json_file(self, json_file_path):
- """ Save this instance to a json file."""
+ """Save this instance to a json file."""
with open(json_file_path, "w", encoding="utf-8") as writer:
writer.write(self.to_json_string())
diff --git a/src/transformers/modeling_flax_utils.py b/src/transformers/modeling_flax_utils.py
index 1331b3ba3997..b32acd0f7d9c 100644
--- a/src/transformers/modeling_flax_utils.py
+++ b/src/transformers/modeling_flax_utils.py
@@ -94,7 +94,7 @@ def __init__(
self.key = PRNGKey(seed)
self.dtype = dtype
- # randomely initialized parameters
+ # randomly initialized parameters
random_params = self.init_weights(self.key, input_shape)
# save required_params as set
diff --git a/src/transformers/modeling_outputs.py b/src/transformers/modeling_outputs.py
index 56d6e3d60409..c352dff787ae 100644
--- a/src/transformers/modeling_outputs.py
+++ b/src/transformers/modeling_outputs.py
@@ -343,7 +343,7 @@ class CausalLMOutputWithPast(ModelOutput):
Language modeling loss (for next-token prediction).
logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`):
Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
- past_key_values (:obj:`tuple(tupel(torch.FloatTensor))`, `optional`, returned when ``use_cache=True`` is passed or when ``config.use_cache=True``):
+ past_key_values (:obj:`tuple(tuple(torch.FloatTensor))`, `optional`, returned when ``use_cache=True`` is passed or when ``config.use_cache=True``):
Tuple of :obj:`tuple(torch.FloatTensor)` of length :obj:`config.n_layers`, with each tuple having 2 tensors
of shape :obj:`(batch_size, num_heads, sequence_length, embed_size_per_head)`)
@@ -423,7 +423,7 @@ class SequenceClassifierOutputWithPast(ModelOutput):
Classification (or regression if config.num_labels==1) loss.
logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, config.num_labels)`):
Classification (or regression if config.num_labels==1) scores (before SoftMax).
- past_key_values (:obj:`tuple(tupel(torch.FloatTensor))`, `optional`, returned when ``use_cache=True`` is passed or when ``config.use_cache=True``):
+ past_key_values (:obj:`tuple(tuple(torch.FloatTensor))`, `optional`, returned when ``use_cache=True`` is passed or when ``config.use_cache=True``):
Tuple of :obj:`tuple(torch.FloatTensor)` of length :obj:`config.n_layers`, with each tuple having 2 tensors
of shape :obj:`(batch_size, num_heads, sequence_length, embed_size_per_head)`)
diff --git a/src/transformers/modeling_tf_outputs.py b/src/transformers/modeling_tf_outputs.py
index 4c98106e3045..fefc65ec9b07 100644
--- a/src/transformers/modeling_tf_outputs.py
+++ b/src/transformers/modeling_tf_outputs.py
@@ -116,6 +116,82 @@ class TFBaseModelOutputWithPast(ModelOutput):
attentions: Optional[Tuple[tf.Tensor]] = None
+@dataclass
+class TFBaseModelOutputWithCrossAttentions(ModelOutput):
+ """
+ Base class for model's outputs, with potential hidden states and attentions.
+
+ Args:
+ last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
+ Sequence of hidden-states at the output of the last layer of the model.
+ hidden_states (:obj:`tuple(tf.FloatTensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
+ Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of
+ shape :obj:`(batch_size, sequence_length, hidden_size)`.
+
+ Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+ attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
+ Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length,
+ sequence_length)`.
+
+ Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+ heads.
+ cross_attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
+ Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length,
+ sequence_length)`.
+
+ Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the
+ weighted average in the cross-attention heads.
+ """
+
+ last_hidden_state: tf.Tensor = None
+ hidden_states: Optional[Tuple[tf.Tensor]] = None
+ attentions: Optional[Tuple[tf.Tensor]] = None
+ cross_attentions: Optional[Tuple[tf.Tensor]] = None
+
+
+@dataclass
+class TFBaseModelOutputWithPastAndCrossAttentions(ModelOutput):
+ """
+ Base class for model's outputs that may also contain a past key/values (to speed up sequential decoding).
+
+ Args:
+ last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
+ Sequence of hidden-states at the output of the last layer of the model.
+
+ If :obj:`past_key_values` is used only the last hidden-state of the sequences of shape :obj:`(batch_size,
+ 1, hidden_size)` is output.
+ past_key_values (:obj:`List[tf.Tensor]`, `optional`, returned when ``use_cache=True`` is passed or when ``config.use_cache=True``):
+ List of :obj:`tf.Tensor` of length :obj:`config.n_layers`, with each tensor of shape :obj:`(2, batch_size,
+ num_heads, sequence_length, embed_size_per_head)`).
+
+ Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see
+ :obj:`past_key_values` input) to speed up sequential decoding.
+ hidden_states (:obj:`tuple(tf.FloatTensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
+ Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of
+ shape :obj:`(batch_size, sequence_length, hidden_size)`.
+
+ Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+ attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
+ Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length,
+ sequence_length)`.
+
+ Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+ heads.
+ cross_attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
+ Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length,
+ sequence_length)`.
+
+ Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the
+ weighted average in the cross-attention heads.
+ """
+
+ last_hidden_state: tf.Tensor = None
+ past_key_values: Optional[List[tf.Tensor]] = None
+ hidden_states: Optional[Tuple[tf.Tensor]] = None
+ attentions: Optional[Tuple[tf.Tensor]] = None
+ cross_attentions: Optional[Tuple[tf.Tensor]] = None
+
+
@dataclass
class TFSeq2SeqModelOutput(ModelOutput):
"""
@@ -145,6 +221,12 @@ class TFSeq2SeqModelOutput(ModelOutput):
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
+ cross_attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
+ Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length,
+ sequence_length)`.
+
+ Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the
+ weighted average in the cross-attention heads.
encoder_last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
@@ -164,6 +246,7 @@ class TFSeq2SeqModelOutput(ModelOutput):
past_key_values: Optional[List[tf.Tensor]] = None
decoder_hidden_states: Optional[Tuple[tf.Tensor]] = None
decoder_attentions: Optional[Tuple[tf.Tensor]] = None
+ cross_attentions: Optional[Tuple[tf.Tensor]] = None
encoder_last_hidden_state: Optional[tf.Tensor] = None
encoder_hidden_states: Optional[Tuple[tf.Tensor]] = None
encoder_attentions: Optional[Tuple[tf.Tensor]] = None
@@ -290,6 +373,12 @@ class TFSeq2SeqLMOutput(ModelOutput):
Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the
self-attention heads.
+ cross_attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
+ Tuple of :obj:`tf.Tensor` (one for each layer) of shape :obj:`(batch_size, num_heads, sequence_length,
+ sequence_length)`.
+
+ Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the
+ weighted average in the cross-attention heads.
encoder_last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Sequence of hidden-states at the output of the last layer of the encoder of the model.
encoder_hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
@@ -310,6 +399,7 @@ class TFSeq2SeqLMOutput(ModelOutput):
past_key_values: Optional[List[tf.Tensor]] = None
decoder_hidden_states: Optional[Tuple[tf.Tensor]] = None
decoder_attentions: Optional[Tuple[tf.Tensor]] = None
+ cross_attentions: Optional[Tuple[tf.Tensor]] = None
encoder_last_hidden_state: Optional[tf.Tensor] = None
encoder_hidden_states: Optional[Tuple[tf.Tensor]] = None
encoder_attentions: Optional[Tuple[tf.Tensor]] = None
diff --git a/src/transformers/modeling_tf_pytorch_utils.py b/src/transformers/modeling_tf_pytorch_utils.py
index fad7e865b9d6..db3d1cf705a4 100644
--- a/src/transformers/modeling_tf_pytorch_utils.py
+++ b/src/transformers/modeling_tf_pytorch_utils.py
@@ -51,7 +51,7 @@ def convert_tf_weight_name_to_pt_weight_name(tf_name, start_prefix_to_remove="")
) # '_._' is replaced by a level separation (can be used to convert TF2.0 lists in PyTorch nn.ModulesList)
tf_name = re.sub(r"//+", "/", tf_name) # Remove empty levels at the end
tf_name = tf_name.split("/") # Convert from TF2.0 '/' separators to PyTorch '.' separators
- # Some weights have a single name withtout "/" such as final_logits_bias in BART
+ # Some weights have a single name without "/" such as final_logits_bias in BART
if len(tf_name) > 1:
tf_name = tf_name[1:] # Remove level zero
diff --git a/src/transformers/modeling_tf_utils.py b/src/transformers/modeling_tf_utils.py
index ea4cff65592b..4bf12af5573c 100644
--- a/src/transformers/modeling_tf_utils.py
+++ b/src/transformers/modeling_tf_utils.py
@@ -659,7 +659,7 @@ def serving(self, inputs):
Args:
inputs (:obj:`Dict[str, tf.Tensor]`):
- The input of the saved model as a dictionnary of tensors.
+ The input of the saved model as a dictionary of tensors.
"""
output = self.call(inputs)
@@ -944,7 +944,7 @@ def _get_resized_lm_head_decoder(self, old_lm_head_decoder, new_num_tokens):
vectors from the end. If not provided or :obj:`None`, just returns None
Return:
- :obj:`tf.Variable`: Pointer to the resized decoder or None if the output embeddings are differents of the
+ :obj:`tf.Variable`: Pointer to the resized decoder or None if the output embeddings are different from the
input ones.
"""
new_lm_head_decoder = old_lm_head_decoder
diff --git a/src/transformers/modeling_utils.py b/src/transformers/modeling_utils.py
index 44513b399451..7b1f477af528 100755
--- a/src/transformers/modeling_utils.py
+++ b/src/transformers/modeling_utils.py
@@ -291,7 +291,7 @@ def get_head_mask(
The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard).
num_hidden_layers (:obj:`int`):
The number of hidden layers in the model.
- is_attention_chunked: (:obj:`bool`, `optional, defaults to :obj:`False`):
+ is_attention_chunked: (:obj:`bool`, `optional`, defaults to :obj:`False`):
Whether or not the attentions scores are computed by chunks or not.
Returns:
@@ -1122,7 +1122,11 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P
import deepspeed
logger.info("Detected DeepSpeed ZeRO-3: activating zero.init() for this model")
- # this immediately partitions the model to avoid the overhead in time and memory copying it on CPU or each GPU first
+ # this immediately partitions the model across all gpus, to avoid the overhead in time
+ # and memory copying it on CPU or each GPU first
+
+ # XXX: param_dict will be added in deepspeed==0.3.16 and probably replaced by deepspeed_config
+ # with deepspeed.zero.Init(param_dict=deepspeed_config()):
with deepspeed.zero.Init():
model = cls(config, *model_args, **model_kwargs)
else:
diff --git a/src/transformers/models/albert/modeling_albert.py b/src/transformers/models/albert/modeling_albert.py
index 21da03fd7a3b..a753c580888f 100755
--- a/src/transformers/models/albert/modeling_albert.py
+++ b/src/transformers/models/albert/modeling_albert.py
@@ -71,7 +71,7 @@
def load_tf_weights_in_albert(model, config, tf_checkpoint_path):
- """ Load tf checkpoints in a pytorch model."""
+ """Load tf checkpoints in a pytorch model."""
try:
import re
diff --git a/src/transformers/models/albert/modeling_tf_albert.py b/src/transformers/models/albert/modeling_tf_albert.py
index 64be5062c864..c750705ee688 100644
--- a/src/transformers/models/albert/modeling_tf_albert.py
+++ b/src/transformers/models/albert/modeling_tf_albert.py
@@ -189,7 +189,7 @@ def call(
class TFAlbertAttention(tf.keras.layers.Layer):
- """ Contains the complete attention sublayer, including both dropouts and layer norm. """
+ """Contains the complete attention sublayer, including both dropouts and layer norm."""
def __init__(self, config: AlbertConfig, **kwargs):
super().__init__(**kwargs)
diff --git a/src/transformers/models/albert/tokenization_albert.py b/src/transformers/models/albert/tokenization_albert.py
index 92c06bbcde63..493a5e145af9 100644
--- a/src/transformers/models/albert/tokenization_albert.py
+++ b/src/transformers/models/albert/tokenization_albert.py
@@ -187,7 +187,7 @@ def preprocess_text(self, inputs):
return outputs
def _tokenize(self, text, sample=False):
- """ Tokenize a string. """
+ """Tokenize a string."""
text = self.preprocess_text(text)
if not sample:
@@ -211,7 +211,7 @@ def _tokenize(self, text, sample=False):
return new_pieces
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.sp_model.PieceToId(token)
def _convert_id_to_token(self, index):
diff --git a/src/transformers/models/auto/modeling_auto.py b/src/transformers/models/auto/modeling_auto.py
index f2770f429648..eca851bbfd78 100644
--- a/src/transformers/models/auto/modeling_auto.py
+++ b/src/transformers/models/auto/modeling_auto.py
@@ -716,7 +716,7 @@
"AutoModelForPreTraining", MODEL_FOR_PRETRAINING_MAPPING, head_doc="pretraining"
)
-# Private on puprose, the public class will add the deprecation warnings.
+# Private on purpose, the public class will add the deprecation warnings.
_AutoModelWithLMHead = auto_class_factory(
"AutoModelWithLMHead", MODEL_WITH_LM_HEAD_MAPPING, head_doc="language modeling"
)
diff --git a/src/transformers/models/auto/modeling_flax_auto.py b/src/transformers/models/auto/modeling_flax_auto.py
index 042612d0a529..12d158d504db 100644
--- a/src/transformers/models/auto/modeling_flax_auto.py
+++ b/src/transformers/models/auto/modeling_flax_auto.py
@@ -103,7 +103,7 @@
)
FlaxAutoModelForSequenceClassification = auto_class_factory(
- "AFlaxutoModelForSequenceClassification",
+ "FlaxAutoModelForSequenceClassification",
FLAX_MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING,
head_doc="sequence classification",
)
diff --git a/src/transformers/models/auto/modeling_tf_auto.py b/src/transformers/models/auto/modeling_tf_auto.py
index 2104bb644299..9bb4b5383f67 100644
--- a/src/transformers/models/auto/modeling_tf_auto.py
+++ b/src/transformers/models/auto/modeling_tf_auto.py
@@ -469,7 +469,7 @@
"TFAutoModelForPreTraining", TF_MODEL_FOR_PRETRAINING_MAPPING, head_doc="pretraining"
)
-# Private on puprose, the public class will add the deprecation warnings.
+# Private on purpose, the public class will add the deprecation warnings.
_TFAutoModelWithLMHead = auto_class_factory(
"TFAutoModelWithLMHead", TF_MODEL_WITH_LM_HEAD_MAPPING, head_doc="language modeling"
)
diff --git a/src/transformers/models/bart/configuration_bart.py b/src/transformers/models/bart/configuration_bart.py
index 0ea94f76b89c..259beda01952 100644
--- a/src/transformers/models/bart/configuration_bart.py
+++ b/src/transformers/models/bart/configuration_bart.py
@@ -171,7 +171,7 @@ def __init__(
self.gradient_checkpointing = gradient_checkpointing
self.scale_embedding = scale_embedding # scale factor will be sqrt(d_model) if True
- # ensure backward compatibilty for BART CNN models
+ # ensure backward compatibility for BART CNN models
if self.forced_bos_token_id is None and kwargs.get("force_bos_token_to_be_generated", False):
self.forced_bos_token_id = self.bos_token_id
warnings.warn(
diff --git a/src/transformers/models/bart/modeling_bart.py b/src/transformers/models/bart/modeling_bart.py
index 523fda70b540..89e078bd9e8e 100755
--- a/src/transformers/models/bart/modeling_bart.py
+++ b/src/transformers/models/bart/modeling_bart.py
@@ -111,7 +111,7 @@ class BartLearnedPositionalEmbedding(nn.Embedding):
def __init__(self, num_embeddings: int, embedding_dim: int):
# Bart is set up so that if padding_idx is specified then offset the embedding ids by 2
- # and adjust num_embeddings appropriately. Other models dont have this hack
+ # and adjust num_embeddings appropriately. Other models don't have this hack
self.offset = 2
super().__init__(num_embeddings + self.offset, embedding_dim)
@@ -236,9 +236,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -582,7 +582,7 @@ def __init_subclass__(self):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
Bart uses the :obj:`eos_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
diff --git a/src/transformers/models/bart/modeling_tf_bart.py b/src/transformers/models/bart/modeling_tf_bart.py
index 5a1fb467ff2a..41f5f9591881 100644
--- a/src/transformers/models/bart/modeling_tf_bart.py
+++ b/src/transformers/models/bart/modeling_tf_bart.py
@@ -30,7 +30,7 @@
)
from ...modeling_tf_outputs import (
TFBaseModelOutput,
- TFBaseModelOutputWithPast,
+ TFBaseModelOutputWithPastAndCrossAttentions,
TFSeq2SeqLMOutput,
TFSeq2SeqModelOutput,
)
@@ -116,7 +116,7 @@ class TFBartLearnedPositionalEmbedding(TFSharedEmbeddings):
def __init__(self, num_embeddings: int, embedding_dim: int, **kwargs):
# Bart is set up so that if padding_idx is specified then offset the embedding ids by 2
- # and adjust num_embeddings appropriately. Other models dont have this hack
+ # and adjust num_embeddings appropriately. Other models don't have this hack
self.offset = 2
super().__init__(num_embeddings + self.offset, embedding_dim, **kwargs)
@@ -365,7 +365,7 @@ def call(
encoder_hidden_states: Optional[tf.Tensor] = None,
encoder_attention_mask: Optional[tf.Tensor] = None,
layer_head_mask: Optional[tf.Tensor] = None,
- encoder_layer_head_mask: Optional[tf.Tensor] = None,
+ cross_attn_layer_head_mask: Optional[tf.Tensor] = None,
past_key_value: Optional[Tuple[tf.Tensor]] = None,
training=False,
) -> Tuple[tf.Tensor, tf.Tensor, Tuple[Tuple[tf.Tensor]]]:
@@ -379,8 +379,8 @@ def call(
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
layer_head_mask (:obj:`tf.Tensor`): mask for attention heads in a given layer of size
`(decoder_attention_heads,)`
- encoder_layer_head_mask (:obj:`tf.Tensor`): mask for encoder attention heads in a given layer of size
- `(encoder_attention_heads,)`
+ cross_attn_layer_head_mask (:obj:`tf.Tensor`): mask for heads of the cross-attention module.
+ `(decoder_attention_heads,)`
past_key_value (:obj:`Tuple(tf.Tensor)`): cached past key and value projection states
"""
residual = hidden_states
@@ -401,16 +401,17 @@ def call(
# Cross-Attention Block
cross_attn_present_key_value = None
+ cross_attn_weights = None
if encoder_hidden_states is not None:
residual = hidden_states
# cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
- hidden_states, _, cross_attn_present_key_value = self.encoder_attn(
+ hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
hidden_states=hidden_states,
key_value_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
- layer_head_mask=encoder_layer_head_mask,
+ layer_head_mask=cross_attn_layer_head_mask,
past_key_value=cross_attn_past_key_value,
)
hidden_states = self.dropout(hidden_states, training=training)
@@ -432,6 +433,7 @@ def call(
return (
hidden_states,
self_attn_weights,
+ cross_attn_weights,
present_key_value,
)
@@ -557,7 +559,7 @@ def serving(self, inputs):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
Bart uses the :obj:`eos_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -572,7 +574,7 @@ def serving(self, inputs):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -580,6 +582,12 @@ def serving(self, inputs):
- 1 indicates the head is **not masked**,
- 0 indicates the head is **masked**.
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
encoder_outputs (:obj:`tf.FloatTensor`, `optional`):
hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
of shape :obj:`(batch_size, sequence_length, hidden_size)` is a sequence of
@@ -677,7 +685,7 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
@@ -814,7 +822,7 @@ def call(
encoder_hidden_states=None,
encoder_attention_mask=None,
head_mask=None,
- encoder_head_mask=None,
+ cross_attn_head_mask=None,
past_key_values=None,
use_cache=None,
output_attentions=None,
@@ -856,14 +864,13 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
- encoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional`):
- Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention
- on hidden heads. Mask values selected in ``[0, 1]``:
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
past_key_values (:obj:`Tuple[Tuple[tf.Tensor]]` of length :obj:`config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up
@@ -894,7 +901,7 @@ def call(
encoder_hidden_states=encoder_hidden_states,
encoder_attention_mask=encoder_attention_mask,
head_mask=head_mask,
- encoder_head_mask=encoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
inputs_embeds=inputs_embeds,
past_key_values=past_key_values,
use_cache=use_cache,
@@ -949,17 +956,19 @@ def call(
# decoder layers
all_hidden_states = () if inputs["output_hidden_states"] else None
all_self_attns = () if inputs["output_attentions"] else None
+ all_cross_attns = () if (inputs["output_attentions"] and inputs["encoder_hidden_states"] is not None) else None
present_key_values = () if inputs["use_cache"] else None
- # check if head_mask has a correct number of layers specified if desired
+ # check if head_mask and cross_attn_head_mask have a correct number of layers specified if desired
# The tf.debugging asserts are not compliant with XLA then they
# have to be disabled in other modes than eager.
- if inputs["head_mask"] is not None and tf.executing_eagerly():
- tf.debugging.assert_equal(
- shape_list(inputs["head_mask"])[0],
- len(self.layers),
- message=f"The head_mask should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs['head_mask'])[0]}.",
- )
+ for attn_mask in ["head_mask", "cross_attn_head_mask"]:
+ if inputs[attn_mask] is not None and tf.executing_eagerly():
+ tf.debugging.assert_equal(
+ shape_list(inputs[attn_mask])[0],
+ len(self.layers),
+ message=f"The {attn_mask} should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs[attn_mask])[0]}.",
+ )
for idx, decoder_layer in enumerate(self.layers):
# add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
@@ -973,14 +982,14 @@ def call(
past_key_value = inputs["past_key_values"][idx] if inputs["past_key_values"] is not None else None
- hidden_states, layer_self_attn, present_key_value = decoder_layer(
+ hidden_states, layer_self_attn, layer_cross_attn, present_key_value = decoder_layer(
hidden_states,
attention_mask=combined_attention_mask,
encoder_hidden_states=inputs["encoder_hidden_states"],
encoder_attention_mask=inputs["encoder_attention_mask"],
layer_head_mask=inputs["head_mask"][idx] if inputs["head_mask"] is not None else None,
- encoder_layer_head_mask=inputs["encoder_head_mask"][idx]
- if inputs["encoder_head_mask"] is not None
+ cross_attn_layer_head_mask=inputs["cross_attn_head_mask"][idx]
+ if inputs["cross_attn_head_mask"] is not None
else None,
past_key_value=past_key_value,
)
@@ -991,23 +1000,30 @@ def call(
if inputs["output_attentions"]:
all_self_attns += (layer_self_attn,)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns += (layer_cross_attn,)
+
if inputs["output_hidden_states"]:
all_hidden_states += (hidden_states,)
if inputs["output_attentions"]:
all_self_attns = list(all_self_attns)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns = list(all_cross_attns)
+
if inputs["use_cache"]:
present_key_values = (inputs["encoder_hidden_states"], present_key_values)
if not inputs["return_dict"]:
- return hidden_states, present_key_values, all_hidden_states, all_self_attns
+ return hidden_states, present_key_values, all_hidden_states, all_self_attns, all_cross_attns
else:
- return TFBaseModelOutputWithPast(
+ return TFBaseModelOutputWithPastAndCrossAttentions(
last_hidden_state=hidden_states,
past_key_values=present_key_values,
hidden_states=all_hidden_states,
attentions=all_self_attns,
+ cross_attentions=all_cross_attns,
)
@@ -1057,6 +1073,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1077,6 +1094,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1131,7 +1149,7 @@ def call(
encoder_hidden_states=inputs["encoder_outputs"][0],
encoder_attention_mask=inputs["attention_mask"],
head_mask=inputs["decoder_head_mask"],
- encoder_head_mask=inputs["head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["decoder_inputs_embeds"],
use_cache=inputs["use_cache"],
@@ -1149,6 +1167,7 @@ def call(
past_key_values=decoder_outputs.past_key_values,
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
+ cross_attentions=decoder_outputs.cross_attentions,
encoder_last_hidden_state=inputs["encoder_outputs"].last_hidden_state,
encoder_hidden_states=inputs["encoder_outputs"].hidden_states,
encoder_attentions=inputs["encoder_outputs"].attentions,
@@ -1189,6 +1208,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1209,6 +1229,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1228,6 +1249,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
encoder_outputs=inputs["encoder_outputs"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
@@ -1245,6 +1267,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1253,6 +1276,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
@@ -1309,6 +1333,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[TFBaseModelOutput] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1339,6 +1364,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1372,6 +1398,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
decoder_inputs_embeds=inputs["decoder_inputs_embeds"],
@@ -1394,6 +1421,7 @@ def call(
past_key_values=outputs.past_key_values, # index 1 of d outputs
decoder_hidden_states=outputs.decoder_hidden_states, # index 2 of d outputs
decoder_attentions=outputs.decoder_attentions, # index 3 of d outputs
+ cross_attentions=outputs.cross_attentions, # index 4 of d outputs
encoder_last_hidden_state=outputs.encoder_last_hidden_state, # index 0 of encoder outputs
encoder_hidden_states=outputs.encoder_hidden_states, # 1 of e out
encoder_attentions=outputs.encoder_attentions, # 2 of e out
@@ -1403,6 +1431,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1411,6 +1440,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
diff --git a/src/transformers/models/barthez/tokenization_barthez.py b/src/transformers/models/barthez/tokenization_barthez.py
index 641cc80c1da8..95d64cfa28d1 100644
--- a/src/transformers/models/barthez/tokenization_barthez.py
+++ b/src/transformers/models/barthez/tokenization_barthez.py
@@ -223,7 +223,7 @@ def _tokenize(self, text):
return self.sp_model.EncodeAsPieces(text)
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
if token in self.fairseq_tokens_to_ids:
return self.fairseq_tokens_to_ids[token]
spm_id = self.sp_model.PieceToId(token)
diff --git a/src/transformers/models/bert/modeling_bert.py b/src/transformers/models/bert/modeling_bert.py
index a1176f3a4ad3..34dd5329beba 100755
--- a/src/transformers/models/bert/modeling_bert.py
+++ b/src/transformers/models/bert/modeling_bert.py
@@ -703,7 +703,7 @@ class BertPreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/bert/tokenization_bert.py b/src/transformers/models/bert/tokenization_bert.py
index fbb2cfc02950..897fb3276187 100644
--- a/src/transformers/models/bert/tokenization_bert.py
+++ b/src/transformers/models/bert/tokenization_bert.py
@@ -233,7 +233,7 @@ def _tokenize(self, text):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.vocab.get(token, self.vocab.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -241,7 +241,7 @@ def _convert_id_to_token(self, index):
return self.ids_to_tokens.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = " ".join(tokens).replace(" ##", "").strip()
return out_string
diff --git a/src/transformers/models/bert_generation/modeling_bert_generation.py b/src/transformers/models/bert_generation/modeling_bert_generation.py
index 6f366c7f4243..dad2d1ceceb7 100755
--- a/src/transformers/models/bert_generation/modeling_bert_generation.py
+++ b/src/transformers/models/bert_generation/modeling_bert_generation.py
@@ -177,7 +177,7 @@ class BertGenerationPreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/bert_generation/tokenization_bert_generation.py b/src/transformers/models/bert_generation/tokenization_bert_generation.py
index 42b5fcac8eb8..795d5f504c22 100644
--- a/src/transformers/models/bert_generation/tokenization_bert_generation.py
+++ b/src/transformers/models/bert_generation/tokenization_bert_generation.py
@@ -119,7 +119,7 @@ def _tokenize(self, text, sample=False):
return pieces
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.sp_model.piece_to_id(token)
def _convert_id_to_token(self, index):
@@ -128,7 +128,7 @@ def _convert_id_to_token(self, index):
return token
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = self.sp_model.decode_pieces(tokens)
return out_string
diff --git a/src/transformers/models/bert_japanese/tokenization_bert_japanese.py b/src/transformers/models/bert_japanese/tokenization_bert_japanese.py
index 242a75e702c5..be62e92e059e 100644
--- a/src/transformers/models/bert_japanese/tokenization_bert_japanese.py
+++ b/src/transformers/models/bert_japanese/tokenization_bert_japanese.py
@@ -304,7 +304,7 @@ def tokenize(self, text, never_split=None, **kwargs):
class CharacterTokenizer:
- """Runs Character tokenziation."""
+ """Runs Character tokenization."""
def __init__(self, vocab, unk_token, normalize_text=True):
"""
diff --git a/src/transformers/models/bertweet/tokenization_bertweet.py b/src/transformers/models/bertweet/tokenization_bertweet.py
index bf110274da1a..76103d051c1b 100644
--- a/src/transformers/models/bertweet/tokenization_bertweet.py
+++ b/src/transformers/models/bertweet/tokenization_bertweet.py
@@ -368,7 +368,7 @@ def normalizeToken(self, token):
return token
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.encoder.get(token, self.encoder.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -376,7 +376,7 @@ def _convert_id_to_token(self, index):
return self.decoder.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = " ".join(tokens).replace("@@ ", "").strip()
return out_string
@@ -451,7 +451,7 @@ def add_from_file(self, f):
the class Tokenizer.
4. When instantiating Tokenizer objects, there is a single option: preserve_case. By default, it is set to True. If it
- is set to False, then the tokenizer will downcase everything except for emoticons.
+ is set to False, then the tokenizer will lowercase everything except for emoticons.
"""
diff --git a/src/transformers/models/big_bird/modeling_big_bird.py b/src/transformers/models/big_bird/modeling_big_bird.py
index 5b5d96b4e9b9..a886e57d1f95 100755
--- a/src/transformers/models/big_bird/modeling_big_bird.py
+++ b/src/transformers/models/big_bird/modeling_big_bird.py
@@ -41,7 +41,6 @@
CausalLMOutputWithCrossAttentions,
MaskedLMOutput,
MultipleChoiceModelOutput,
- QuestionAnsweringModelOutput,
SequenceClassifierOutput,
TokenClassifierOutput,
)
@@ -485,7 +484,7 @@ def forward(
@staticmethod
def torch_bmm_nd(inp_1, inp_2, ndim=None):
- """ Fast nd matrix multiplication """
+ """Fast nd matrix multiplication"""
# faster replacement of torch.einsum ("bhqk,bhkd->bhqd")
return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
inp_1.shape[: ndim - 2] + (inp_1.shape[ndim - 2], inp_2.shape[ndim - 1])
@@ -493,7 +492,7 @@ def torch_bmm_nd(inp_1, inp_2, ndim=None):
@staticmethod
def torch_bmm_nd_transpose(inp_1, inp_2, ndim=None):
- """ Fast nd matrix multiplication with transpose """
+ """Fast nd matrix multiplication with transpose"""
# faster replacement of torch.einsum (bhqd,bhkd->bhqk)
return torch.bmm(
inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:]).transpose(1, 2)
@@ -725,7 +724,7 @@ def bigbird_block_sparse_attention(
band_product, dim=-1
) # [bsz, n_heads, from_seq_len//from_block_size-4, from_block_size, (5+n_rand_blocks)*to_block_size]
- # contibution of sliding keys
+ # contribution of sliding keys
# [bsz, n_heads, m//from_block_size-4, from_block_size, 3*to_block_size] x [bsz, n_heads, from_seq_len//from_block_size-4, 3*to_block_size, -1]
context_layer = self.torch_bmm_nd(
attn_weights[:, :, :, :, to_block_size : 4 * to_block_size], exp_blocked_value_matrix, ndim=5
@@ -877,7 +876,7 @@ def bigbird_block_sparse_attention(
attn_probs_view[:, :, q_idx, :, q_idx : q_idx + 3, :] = right_slice.view(
bsz, n_heads, from_block_size, 3, to_block_size
) # inner_band_product
- # global keys (correspomding to 1st key block)
+ # global keys (corresponding to 1st key block)
attention_probs[:, :, 2 * from_block_size : -2 * from_block_size, :to_block_size] = attn_weights[
:, :, :, :, :to_block_size
].view(
@@ -947,7 +946,7 @@ def bigbird_block_sparse_attention(
@staticmethod
def torch_gather_b2(params, indices):
- # this operation is equilvalent to tf.gather when batch_dims=2
+ # this operation is equivalent to tf.gather when batch_dims=2
if params.shape[:2] != indices.shape[:2]:
raise ValueError(
@@ -1055,7 +1054,7 @@ def _bigbird_block_rand_mask(
to_block_size: int. size of block in to sequence.
num_rand_blocks: int. Number of random chunks per row.
last_idx: if -1 then num_rand_blocks blocks chosen anywhere in to sequence,
- if positive then num_rand_blocks blocks choosen only upto last_idx.
+ if positive then num_rand_blocks blocks chosen only up to last_idx.
Returns:
adjacency list of size from_seq_length//from_block_size-2 by num_rand_blocks
@@ -1150,7 +1149,7 @@ def _bigbird_block_rand_mask_with_head(
plan_block_length = np.array(plan_from_length) // from_block_size
# till when to follow plan
max_plan_idx = plan_from_length.index(from_seq_length)
- # Random Attention adjajency list
+ # Random Attention adjacency list
rand_attn = [
np.zeros((num_blocks, np.sum(plan_num_rand_blocks[: max_plan_idx + 1])), dtype=np.int32)
for i in range(num_heads)
@@ -1247,8 +1246,8 @@ def _get_single_block_row_attention(
Args:
block_id: int. block id of row.
- to_start_block_id: int. random attention coloum start id.
- to_end_block_id: int. random attention coloum end id.
+ to_start_block_id: int. random attention column start id.
+ to_end_block_id: int. random attention column end id.
num_rand_blocks: int. number of random blocks to be selected.
window_block_left: int. number of blocks of window to left of a block.
window_block_right: int. number of blocks of window to right of a block.
@@ -1744,7 +1743,7 @@ class BigBirdPreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
@@ -1826,7 +1825,7 @@ def _init_weights(self, module):
@dataclass
class BigBirdForPreTrainingOutput(ModelOutput):
"""
- Output type of :class:`~transformers.BigBirdtForPreTraining`.
+ Output type of :class:`~transformers.BigBirdForPreTraining`.
Args:
loss (`optional`, returned when ``labels`` is provided, ``torch.FloatTensor`` of shape :obj:`(1,)`):
@@ -1857,6 +1856,41 @@ class BigBirdForPreTrainingOutput(ModelOutput):
attentions: Optional[Tuple[torch.FloatTensor]] = None
+@dataclass
+class BigBirdForQuestionAnsweringModelOutput(ModelOutput):
+ """
+ Base class for outputs of question answering models.
+
+ Args:
+ loss (:obj:`torch.FloatTensor` of shape :obj:`(1,)`, `optional`, returned when :obj:`labels` is provided):
+ Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.
+ start_logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`):
+ Span-start scores (before SoftMax).
+ end_logits (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`):
+ Span-end scores (before SoftMax).
+ pooler_output (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, 1)`):
+ pooler output from BigBigModel
+ hidden_states (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
+ Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
+ of shape :obj:`(batch_size, sequence_length, hidden_size)`.
+
+ Hidden-states of the model at the output of each layer plus the initial embedding outputs.
+ attentions (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
+ Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape :obj:`(batch_size, num_heads,
+ sequence_length, sequence_length)`.
+
+ Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
+ heads.
+ """
+
+ loss: Optional[torch.FloatTensor] = None
+ start_logits: torch.FloatTensor = None
+ end_logits: torch.FloatTensor = None
+ pooler_output: torch.FloatTensor = None
+ hidden_states: Optional[Tuple[torch.FloatTensor]] = None
+ attentions: Optional[Tuple[torch.FloatTensor]] = None
+
+
@add_start_docstrings(
"The bare BigBird Model transformer outputting raw hidden-states without any specific head on top.",
BIG_BIRD_START_DOCSTRING,
@@ -2852,14 +2886,14 @@ def forward(self, encoder_output):
BIG_BIRD_START_DOCSTRING,
)
class BigBirdForQuestionAnswering(BigBirdPreTrainedModel):
- def __init__(self, config):
+ def __init__(self, config, add_pooling_layer=False):
super().__init__(config)
config.num_labels = 2
self.num_labels = config.num_labels
self.sep_token_id = config.sep_token_id
- self.bert = BigBirdModel(config, add_pooling_layer=False)
+ self.bert = BigBirdModel(config, add_pooling_layer=add_pooling_layer)
self.qa_classifier = BigBirdForQuestionAnsweringHead(config)
self.init_weights()
@@ -2868,7 +2902,7 @@ def __init__(self, config):
@add_code_sample_docstrings(
tokenizer_class=_TOKENIZER_FOR_DOC,
checkpoint="google/bigbird-base-trivia-itc",
- output_type=QuestionAnsweringModelOutput,
+ output_type=BigBirdForQuestionAnsweringModelOutput,
config_class=_CONFIG_FOR_DOC,
)
def forward(
@@ -2907,7 +2941,7 @@ def forward(
logits_mask = None
if question_lengths is not None:
- # setting lengths logits to `-infi`
+ # setting lengths logits to `-inf`
logits_mask = self.prepare_question_mask(question_lengths, seqlen)
if token_type_ids is None:
token_type_ids = (~logits_mask).long()
@@ -2958,10 +2992,11 @@ def forward(
output = (start_logits, end_logits) + outputs[2:]
return ((total_loss,) + output) if total_loss is not None else output
- return QuestionAnsweringModelOutput(
+ return BigBirdForQuestionAnsweringModelOutput(
loss=total_loss,
start_logits=start_logits,
end_logits=end_logits,
+ pooler_output=outputs.pooler_output,
hidden_states=outputs.hidden_states,
attentions=outputs.attentions,
)
diff --git a/src/transformers/models/big_bird/tokenization_big_bird.py b/src/transformers/models/big_bird/tokenization_big_bird.py
index a2c3dd023ab1..e3e5a93f6da7 100644
--- a/src/transformers/models/big_bird/tokenization_big_bird.py
+++ b/src/transformers/models/big_bird/tokenization_big_bird.py
@@ -149,7 +149,7 @@ def _tokenize(self, text, sample=False):
return pieces
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.sp_model.piece_to_id(token)
def _convert_id_to_token(self, index):
@@ -158,7 +158,7 @@ def _convert_id_to_token(self, index):
return token
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = self.sp_model.decode_pieces(tokens)
return out_string
diff --git a/src/transformers/models/blenderbot/modeling_blenderbot.py b/src/transformers/models/blenderbot/modeling_blenderbot.py
index dc18993c52fe..461084ea73e6 100755
--- a/src/transformers/models/blenderbot/modeling_blenderbot.py
+++ b/src/transformers/models/blenderbot/modeling_blenderbot.py
@@ -237,9 +237,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -546,7 +546,7 @@ def dummy_inputs(self):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
Blenderbot uses the :obj:`bos_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
diff --git a/src/transformers/models/blenderbot/modeling_tf_blenderbot.py b/src/transformers/models/blenderbot/modeling_tf_blenderbot.py
index 42bcad541121..687cd2c7b81f 100644
--- a/src/transformers/models/blenderbot/modeling_tf_blenderbot.py
+++ b/src/transformers/models/blenderbot/modeling_tf_blenderbot.py
@@ -32,7 +32,7 @@
)
from ...modeling_tf_outputs import (
TFBaseModelOutput,
- TFBaseModelOutputWithPast,
+ TFBaseModelOutputWithPastAndCrossAttentions,
TFSeq2SeqLMOutput,
TFSeq2SeqModelOutput,
)
@@ -370,7 +370,7 @@ def call(
encoder_hidden_states: Optional[tf.Tensor] = None,
encoder_attention_mask: Optional[tf.Tensor] = None,
layer_head_mask: Optional[tf.Tensor] = None,
- encoder_layer_head_mask: Optional[tf.Tensor] = None,
+ cross_attn_layer_head_mask: Optional[tf.Tensor] = None,
past_key_value: Optional[Tuple[tf.Tensor]] = None,
training=False,
) -> Tuple[tf.Tensor, tf.Tensor, Tuple[Tuple[tf.Tensor]]]:
@@ -384,8 +384,8 @@ def call(
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
layer_head_mask (:obj:`tf.Tensor`): mask for attention heads in a given layer of size
`(decoder_attention_heads,)`
- encoder_layer_head_mask (:obj:`tf.Tensor`): mask for encoder attention heads in a given layer of size
- `(encoder_attention_heads,)`
+ cross_attn_layer_head_mask (:obj:`tf.Tensor`): mask for heads of the cross-attention module.
+ `(decoder_attention_heads,)`
past_key_value (:obj:`Tuple(tf.Tensor)`): cached past key and value projection states
"""
residual = hidden_states
@@ -406,17 +406,18 @@ def call(
# Cross-Attention Block
cross_attn_present_key_value = None
+ cross_attn_weights = None
if encoder_hidden_states is not None:
residual = hidden_states
hidden_states = self.encoder_attn_layer_norm(hidden_states)
# cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
- hidden_states, _, cross_attn_present_key_value = self.encoder_attn(
+ hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
hidden_states=hidden_states,
key_value_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
- layer_head_mask=encoder_layer_head_mask,
+ layer_head_mask=cross_attn_layer_head_mask,
past_key_value=cross_attn_past_key_value,
)
hidden_states = self.dropout(hidden_states, training=training)
@@ -437,6 +438,7 @@ def call(
return (
hidden_states,
self_attn_weights,
+ cross_attn_weights,
present_key_value,
)
@@ -558,7 +560,7 @@ def serving(self, inputs):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
Blenderbot uses the :obj:`bos_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -569,7 +571,7 @@ def serving(self, inputs):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -577,6 +579,12 @@ def serving(self, inputs):
- 1 indicates the head is **not masked**,
- 0 indicates the head is **masked**.
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
encoder_outputs (:obj:`tf.FloatTensor`, `optional`):
hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
of shape :obj:`(batch_size, sequence_length, hidden_size)` is a sequence of
@@ -674,7 +682,7 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
@@ -818,7 +826,7 @@ def call(
encoder_hidden_states=None,
encoder_attention_mask=None,
head_mask=None,
- encoder_head_mask=None,
+ cross_attn_head_mask=None,
past_key_values=None,
use_cache=None,
output_attentions=None,
@@ -860,14 +868,13 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
- encoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional`):
- Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention
- on hidden heads. Mask values selected in ``[0, 1]``:
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
past_key_values (:obj:`Tuple[Tuple[tf.Tensor]]` of length :obj:`config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up
@@ -904,7 +911,7 @@ def call(
encoder_hidden_states=encoder_hidden_states,
encoder_attention_mask=encoder_attention_mask,
head_mask=head_mask,
- encoder_head_mask=encoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
inputs_embeds=inputs_embeds,
past_key_values=past_key_values,
use_cache=use_cache,
@@ -957,19 +964,21 @@ def call(
hidden_states = self.dropout(hidden_states, training=inputs["training"])
# decoder layers
- all_hidden_states = ()
- all_self_attns = ()
- present_key_values = ()
+ all_hidden_states = () if inputs["output_hidden_states"] else None
+ all_self_attns = () if inputs["output_attentions"] else None
+ all_cross_attns = () if (inputs["output_attentions"] and inputs["encoder_hidden_states"] is not None) else None
+ present_key_values = () if inputs["use_cache"] else None
- # check if head_mask has a correct number of layers specified if desired
+ # check if head_mask and cross_attn_head_mask have a correct number of layers specified if desired
# The tf.debugging asserts are not compliant with XLA then they
# have to be disabled in other modes than eager.
- if inputs["head_mask"] is not None and tf.executing_eagerly():
- tf.debugging.assert_equal(
- shape_list(inputs["head_mask"])[0],
- len(self.layers),
- message=f"The head_mask should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs['head_mask'])[0]}.",
- )
+ for attn_mask in ["head_mask", "cross_attn_head_mask"]:
+ if inputs[attn_mask] is not None and tf.executing_eagerly():
+ tf.debugging.assert_equal(
+ shape_list(inputs[attn_mask])[0],
+ len(self.layers),
+ message=f"The {attn_mask} should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs[attn_mask])[0]}.",
+ )
for idx, decoder_layer in enumerate(self.layers):
# add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
@@ -982,14 +991,14 @@ def call(
past_key_value = inputs["past_key_values"][idx] if inputs["past_key_values"] is not None else None
- hidden_states, layer_self_attn, present_key_value = decoder_layer(
+ hidden_states, layer_self_attn, layer_cross_attn, present_key_value = decoder_layer(
hidden_states,
attention_mask=combined_attention_mask,
encoder_hidden_states=inputs["encoder_hidden_states"],
encoder_attention_mask=inputs["encoder_attention_mask"],
layer_head_mask=inputs["head_mask"][idx] if inputs["head_mask"] is not None else None,
- encoder_layer_head_mask=inputs["encoder_head_mask"][idx]
- if inputs["encoder_head_mask"] is not None
+ cross_attn_layer_head_mask=inputs["cross_attn_head_mask"][idx]
+ if inputs["cross_attn_head_mask"] is not None
else None,
past_key_value=past_key_value,
)
@@ -1000,25 +1009,32 @@ def call(
if inputs["output_attentions"]:
all_self_attns += (layer_self_attn,)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns += (layer_cross_attn,)
+
hidden_states = self.layer_norm(hidden_states)
if inputs["output_hidden_states"]:
all_hidden_states += (hidden_states,)
- else:
- all_hidden_states = None
- all_self_attns = list(all_self_attns) if inputs["output_attentions"] else None
+ if inputs["output_attentions"]:
+ all_self_attns = list(all_self_attns)
+
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns = list(all_cross_attns)
- present_key_values = (encoder_hidden_states, present_key_values) if inputs["use_cache"] else None
+ if inputs["use_cache"]:
+ present_key_values = (inputs["encoder_hidden_states"], present_key_values)
if not inputs["return_dict"]:
- return hidden_states, present_key_values, all_hidden_states, all_self_attns
+ return hidden_states, present_key_values, all_hidden_states, all_self_attns, all_cross_attns
else:
- return TFBaseModelOutputWithPast(
+ return TFBaseModelOutputWithPastAndCrossAttentions(
last_hidden_state=hidden_states,
past_key_values=present_key_values,
hidden_states=all_hidden_states,
attentions=all_self_attns,
+ cross_attentions=all_cross_attns,
)
@@ -1065,6 +1081,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1085,6 +1102,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1131,7 +1149,7 @@ def call(
encoder_hidden_states=inputs["encoder_outputs"][0],
encoder_attention_mask=inputs["attention_mask"],
head_mask=inputs["decoder_head_mask"],
- encoder_head_mask=inputs["head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["decoder_inputs_embeds"],
use_cache=inputs["use_cache"],
@@ -1149,6 +1167,7 @@ def call(
past_key_values=decoder_outputs.past_key_values,
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
+ cross_attentions=decoder_outputs.cross_attentions,
encoder_last_hidden_state=inputs["encoder_outputs"].last_hidden_state,
encoder_hidden_states=inputs["encoder_outputs"].hidden_states,
encoder_attentions=inputs["encoder_outputs"].attentions,
@@ -1199,6 +1218,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1219,6 +1239,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1238,6 +1259,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
encoder_outputs=inputs["encoder_outputs"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
@@ -1256,6 +1278,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1264,6 +1287,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
@@ -1331,6 +1355,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[TFBaseModelOutput] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1361,6 +1386,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1394,6 +1420,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
decoder_inputs_embeds=inputs["decoder_inputs_embeds"],
@@ -1416,6 +1443,7 @@ def call(
past_key_values=outputs.past_key_values, # index 1 of d outputs
decoder_hidden_states=outputs.decoder_hidden_states, # index 2 of d outputs
decoder_attentions=outputs.decoder_attentions, # index 3 of d outputs
+ cross_attentions=outputs.cross_attentions, # index 4 of d outputs
encoder_last_hidden_state=outputs.encoder_last_hidden_state, # index 0 of encoder outputs
encoder_hidden_states=outputs.encoder_hidden_states, # 1 of e out
encoder_attentions=outputs.encoder_attentions, # 2 of e out
@@ -1426,6 +1454,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1434,6 +1463,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
diff --git a/src/transformers/models/blenderbot_small/modeling_blenderbot_small.py b/src/transformers/models/blenderbot_small/modeling_blenderbot_small.py
index 167f9a9bac7b..d32a98ec73c8 100755
--- a/src/transformers/models/blenderbot_small/modeling_blenderbot_small.py
+++ b/src/transformers/models/blenderbot_small/modeling_blenderbot_small.py
@@ -235,9 +235,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -547,7 +547,7 @@ def dummy_inputs(self):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
BlenderbotSmall uses the :obj:`bos_token_id` as the starting token for :obj:`decoder_input_ids` generation.
If :obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
diff --git a/src/transformers/models/blenderbot_small/modeling_tf_blenderbot_small.py b/src/transformers/models/blenderbot_small/modeling_tf_blenderbot_small.py
index 85ae9e9a4a12..49bc59757b2c 100644
--- a/src/transformers/models/blenderbot_small/modeling_tf_blenderbot_small.py
+++ b/src/transformers/models/blenderbot_small/modeling_tf_blenderbot_small.py
@@ -30,7 +30,7 @@
)
from ...modeling_tf_outputs import (
TFBaseModelOutput,
- TFBaseModelOutputWithPast,
+ TFBaseModelOutputWithPastAndCrossAttentions,
TFSeq2SeqLMOutput,
TFSeq2SeqModelOutput,
)
@@ -369,7 +369,7 @@ def call(
encoder_hidden_states: Optional[tf.Tensor] = None,
encoder_attention_mask: Optional[tf.Tensor] = None,
layer_head_mask: Optional[tf.Tensor] = None,
- encoder_layer_head_mask: Optional[tf.Tensor] = None,
+ cross_attn_layer_head_mask: Optional[tf.Tensor] = None,
past_key_value: Optional[Tuple[tf.Tensor]] = None,
training=False,
) -> Tuple[tf.Tensor, tf.Tensor, Tuple[Tuple[tf.Tensor]]]:
@@ -383,8 +383,8 @@ def call(
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
layer_head_mask (:obj:`tf.Tensor`): mask for attention heads in a given layer of size
`(decoder_attention_heads,)`
- encoder_layer_head_mask (:obj:`tf.Tensor`): mask for encoder attention heads in a given layer of size
- `(encoder_attention_heads,)`
+ cross_attn_layer_head_mask (:obj:`tf.Tensor`): mask for heads of the cross-attention module.
+ `(decoder_attention_heads,)`
past_key_value (:obj:`Tuple(tf.Tensor)`): cached past key and value projection states
"""
residual = hidden_states
@@ -405,16 +405,17 @@ def call(
# Cross-Attention Block
cross_attn_present_key_value = None
+ cross_attn_weights = None
if encoder_hidden_states is not None:
residual = hidden_states
# cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
- hidden_states, _, cross_attn_present_key_value = self.encoder_attn(
+ hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
hidden_states=hidden_states,
key_value_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
- layer_head_mask=encoder_layer_head_mask,
+ layer_head_mask=cross_attn_layer_head_mask,
past_key_value=cross_attn_past_key_value,
)
hidden_states = self.dropout(hidden_states, training=training)
@@ -436,6 +437,7 @@ def call(
return (
hidden_states,
self_attn_weights,
+ cross_attn_weights,
present_key_value,
)
@@ -563,7 +565,7 @@ def serving(self, inputs):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
BlenderbotSmall uses the :obj:`bos_token_id` as the starting token for :obj:`decoder_input_ids` generation.
If :obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -574,7 +576,7 @@ def serving(self, inputs):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -582,6 +584,12 @@ def serving(self, inputs):
- 1 indicates the head is **not masked**,
- 0 indicates the head is **masked**.
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
encoder_outputs (:obj:`tf.FloatTensor`, `optional`):
hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
of shape :obj:`(batch_size, sequence_length, hidden_size)` is a sequence of
@@ -679,7 +687,7 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
@@ -823,7 +831,7 @@ def call(
encoder_hidden_states=None,
encoder_attention_mask=None,
head_mask=None,
- encoder_head_mask=None,
+ cross_attn_head_mask=None,
past_key_values=None,
use_cache=None,
output_attentions=None,
@@ -865,14 +873,13 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
- encoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional`):
- Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention
- on hidden heads. Mask values selected in ``[0, 1]``:
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
past_key_values (:obj:`Tuple[Tuple[tf.Tensor]]` of length :obj:`config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up
@@ -909,7 +916,7 @@ def call(
encoder_hidden_states=encoder_hidden_states,
encoder_attention_mask=encoder_attention_mask,
head_mask=head_mask,
- encoder_head_mask=encoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
inputs_embeds=inputs_embeds,
past_key_values=past_key_values,
use_cache=use_cache,
@@ -960,19 +967,21 @@ def call(
hidden_states = self.dropout(hidden_states, training=inputs["training"])
# decoder layers
- all_hidden_states = ()
- all_self_attns = ()
- present_key_values = ()
+ all_hidden_states = () if inputs["output_hidden_states"] else None
+ all_self_attns = () if inputs["output_attentions"] else None
+ all_cross_attns = () if (inputs["output_attentions"] and inputs["encoder_hidden_states"] is not None) else None
+ present_key_values = () if inputs["use_cache"] else None
- # check if head_mask has a correct number of layers specified if desired
+ # check if head_mask and cross_attn_head_mask have a correct number of layers specified if desired
# The tf.debugging asserts are not compliant with XLA then they
# have to be disabled in other modes than eager.
- if inputs["head_mask"] is not None and tf.executing_eagerly():
- tf.debugging.assert_equal(
- shape_list(inputs["head_mask"])[0],
- len(self.layers),
- message=f"The head_mask should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs['head_mask'])[0]}.",
- )
+ for attn_mask in ["head_mask", "cross_attn_head_mask"]:
+ if inputs[attn_mask] is not None and tf.executing_eagerly():
+ tf.debugging.assert_equal(
+ shape_list(inputs[attn_mask])[0],
+ len(self.layers),
+ message=f"The {attn_mask} should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs[attn_mask])[0]}.",
+ )
for idx, decoder_layer in enumerate(self.layers):
# add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
@@ -985,14 +994,14 @@ def call(
past_key_value = inputs["past_key_values"][idx] if inputs["past_key_values"] is not None else None
- hidden_states, layer_self_attn, present_key_value = decoder_layer(
+ hidden_states, layer_self_attn, layer_cross_attn, present_key_value = decoder_layer(
hidden_states,
attention_mask=combined_attention_mask,
encoder_hidden_states=inputs["encoder_hidden_states"],
encoder_attention_mask=inputs["encoder_attention_mask"],
layer_head_mask=inputs["head_mask"][idx] if inputs["head_mask"] is not None else None,
- encoder_layer_head_mask=inputs["encoder_head_mask"][idx]
- if inputs["encoder_head_mask"] is not None
+ cross_attn_layer_head_mask=inputs["cross_attn_head_mask"][idx]
+ if inputs["cross_attn_head_mask"] is not None
else None,
past_key_value=past_key_value,
)
@@ -1003,23 +1012,30 @@ def call(
if inputs["output_attentions"]:
all_self_attns += (layer_self_attn,)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns += (layer_cross_attn,)
+
if inputs["output_hidden_states"]:
all_hidden_states += (hidden_states,)
- else:
- all_hidden_states = None
- all_self_attns = list(all_self_attns) if inputs["output_attentions"] else None
+ if inputs["output_attentions"]:
+ all_self_attns = list(all_self_attns)
+
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns = list(all_cross_attns)
- present_key_values = (encoder_hidden_states, present_key_values) if inputs["use_cache"] else None
+ if inputs["use_cache"]:
+ present_key_values = (inputs["encoder_hidden_states"], present_key_values)
if not inputs["return_dict"]:
- return hidden_states, present_key_values, all_hidden_states, all_self_attns
+ return hidden_states, present_key_values, all_hidden_states, all_self_attns, all_cross_attns
else:
- return TFBaseModelOutputWithPast(
+ return TFBaseModelOutputWithPastAndCrossAttentions(
last_hidden_state=hidden_states,
past_key_values=present_key_values,
hidden_states=all_hidden_states,
attentions=all_self_attns,
+ cross_attentions=all_cross_attns,
)
@@ -1066,6 +1082,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1086,6 +1103,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1132,7 +1150,7 @@ def call(
encoder_hidden_states=inputs["encoder_outputs"][0],
encoder_attention_mask=inputs["attention_mask"],
head_mask=inputs["decoder_head_mask"],
- encoder_head_mask=inputs["head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["decoder_inputs_embeds"],
use_cache=inputs["use_cache"],
@@ -1150,6 +1168,7 @@ def call(
past_key_values=decoder_outputs.past_key_values,
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
+ cross_attentions=decoder_outputs.cross_attentions,
encoder_last_hidden_state=inputs["encoder_outputs"].last_hidden_state,
encoder_hidden_states=inputs["encoder_outputs"].hidden_states,
encoder_attentions=inputs["encoder_outputs"].attentions,
@@ -1187,6 +1206,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1207,6 +1227,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1226,6 +1247,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
encoder_outputs=inputs["encoder_outputs"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
@@ -1244,6 +1266,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1252,6 +1275,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
@@ -1306,6 +1330,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[TFBaseModelOutput] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1336,6 +1361,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1369,6 +1395,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
decoder_inputs_embeds=inputs["decoder_inputs_embeds"],
@@ -1391,6 +1418,7 @@ def call(
past_key_values=outputs.past_key_values, # index 1 of d outputs
decoder_hidden_states=outputs.decoder_hidden_states, # index 2 of d outputs
decoder_attentions=outputs.decoder_attentions, # index 3 of d outputs
+ cross_attentions=outputs.cross_attentions, # index 4 of d outputs
encoder_last_hidden_state=outputs.encoder_last_hidden_state, # index 0 of encoder outputs
encoder_hidden_states=outputs.encoder_hidden_states, # 1 of e out
encoder_attentions=outputs.encoder_attentions, # 2 of e out
@@ -1401,6 +1429,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1409,6 +1438,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
diff --git a/src/transformers/models/blenderbot_small/tokenization_blenderbot_small.py b/src/transformers/models/blenderbot_small/tokenization_blenderbot_small.py
index 1af143f380c8..1b8104e92451 100644
--- a/src/transformers/models/blenderbot_small/tokenization_blenderbot_small.py
+++ b/src/transformers/models/blenderbot_small/tokenization_blenderbot_small.py
@@ -183,7 +183,7 @@ def bpe(self, token: str) -> str:
return " ".join(words)
def _tokenize(self, text: str) -> List[str]:
- """ Split a string into tokens using BPE."""
+ """Split a string into tokens using BPE."""
split_tokens = []
words = re.findall(r"\S+\n?", text)
@@ -193,7 +193,7 @@ def _tokenize(self, text: str) -> List[str]:
return split_tokens
def _convert_token_to_id(self, token: str) -> int:
- """ Converts a token to an id using the vocab. """
+ """Converts a token to an id using the vocab."""
token = token.lower()
return self.encoder.get(token, self.encoder.get(self.unk_token))
@@ -202,7 +202,7 @@ def _convert_id_to_token(self, index: int) -> str:
return self.decoder.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens: List[str]) -> str:
- """ Converts a sequence of tokens in a single string. """
+ """Converts a sequence of tokens in a single string."""
out_string = " ".join(tokens).replace("@@ ", "").strip()
return out_string
diff --git a/src/transformers/models/camembert/tokenization_camembert.py b/src/transformers/models/camembert/tokenization_camembert.py
index 8337d6826cb8..b7bee4e19c49 100644
--- a/src/transformers/models/camembert/tokenization_camembert.py
+++ b/src/transformers/models/camembert/tokenization_camembert.py
@@ -222,7 +222,7 @@ def _tokenize(self, text):
return self.sp_model.EncodeAsPieces(text)
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
if token in self.fairseq_tokens_to_ids:
return self.fairseq_tokens_to_ids[token]
elif self.sp_model.PieceToId(token) == 0:
diff --git a/src/transformers/models/convbert/modeling_convbert.py b/src/transformers/models/convbert/modeling_convbert.py
index 0ededdc83f3f..f597ff178912 100755
--- a/src/transformers/models/convbert/modeling_convbert.py
+++ b/src/transformers/models/convbert/modeling_convbert.py
@@ -238,7 +238,7 @@ class ConvBertPreTrainedModel(PreTrainedModel):
authorized_unexpected_keys = [r"convbert\.embeddings_project\.weight", r"convbert\.embeddings_project\.bias"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/convbert/modeling_tf_convbert.py b/src/transformers/models/convbert/modeling_tf_convbert.py
index ddf33098b25b..f088db5ad168 100644
--- a/src/transformers/models/convbert/modeling_tf_convbert.py
+++ b/src/transformers/models/convbert/modeling_tf_convbert.py
@@ -695,7 +695,7 @@ class TFConvBertPreTrainedModel(TFPreTrainedModel):
Mask to avoid performing attention on padding token indices. Mask values selected in ``[0, 1]``:
- 1 for tokens that are **not masked**,
- - 0 for tokens that are **maked**.
+ - 0 for tokens that are **masked**.
`What are attention masks? <../glossary.html#attention-mask>`__
token_type_ids (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`({0})`, `optional`):
@@ -739,7 +739,7 @@ class TFConvBertPreTrainedModel(TFPreTrainedModel):
@add_start_docstrings(
- "The bare ConvBERT Model transformer outputing raw hidden-states without any specific head on top.",
+ "The bare ConvBERT Model transformer outputting raw hidden-states without any specific head on top.",
CONVBERT_START_DOCSTRING,
)
class TFConvBertModel(TFConvBertPreTrainedModel):
diff --git a/src/transformers/models/ctrl/modeling_ctrl.py b/src/transformers/models/ctrl/modeling_ctrl.py
index edad67dec983..ce9bd80c5929 100644
--- a/src/transformers/models/ctrl/modeling_ctrl.py
+++ b/src/transformers/models/ctrl/modeling_ctrl.py
@@ -683,7 +683,7 @@ def forward(
sequence_lengths = -1
logger.warning(
f"{self.__class__.__name__} will not detect padding tokens in `inputs_embeds`. Results may be "
- f"unexpected if using padding tokens in conjuction with `inputs_embeds.`"
+ f"unexpected if using padding tokens in conjunction with `inputs_embeds.`"
)
pooled_logits = logits[range(batch_size), sequence_lengths]
diff --git a/src/transformers/models/ctrl/tokenization_ctrl.py b/src/transformers/models/ctrl/tokenization_ctrl.py
index d1adb5008728..31ac0637a99b 100644
--- a/src/transformers/models/ctrl/tokenization_ctrl.py
+++ b/src/transformers/models/ctrl/tokenization_ctrl.py
@@ -212,7 +212,7 @@ def _tokenize(self, text):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.encoder.get(token, self.encoder.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -220,7 +220,7 @@ def _convert_id_to_token(self, index):
return self.decoder.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = " ".join(tokens).replace("@@ ", "").strip()
return out_string
diff --git a/src/transformers/models/deberta_v2/modeling_deberta_v2.py b/src/transformers/models/deberta_v2/modeling_deberta_v2.py
index da73997e1a03..03563b02b913 100644
--- a/src/transformers/models/deberta_v2/modeling_deberta_v2.py
+++ b/src/transformers/models/deberta_v2/modeling_deberta_v2.py
@@ -511,7 +511,7 @@ def build_relative_position(query_size, key_size, bucket_size=-1, max_position=-
query_size (int): the length of query
key_size (int): the length of key
bucket_size (int): the size of position bucket
- max_position (int): the maxium allowed absolute positoin
+ max_position (int): the maximum allowed absolute position
Return:
:obj:`torch.LongTensor`: A tensor with shape [1, query_size, key_size]
@@ -698,7 +698,7 @@ def disentangled_attention_bias(self, query_layer, key_layer, relative_pos, rel_
relative_pos = relative_pos.unsqueeze(1)
# bsz x height x query x key
elif relative_pos.dim() != 4:
- raise ValueError(f"Relative postion ids must be of dim 2 or 3 or 4. {relative_pos.dim()}")
+ raise ValueError(f"Relative position ids must be of dim 2 or 3 or 4. {relative_pos.dim()}")
att_span = self.pos_ebd_size
relative_pos = relative_pos.long().to(query_layer.device)
diff --git a/src/transformers/models/deberta_v2/tokenization_deberta_v2.py b/src/transformers/models/deberta_v2/tokenization_deberta_v2.py
index 78509f88d774..ddb77c621b36 100644
--- a/src/transformers/models/deberta_v2/tokenization_deberta_v2.py
+++ b/src/transformers/models/deberta_v2/tokenization_deberta_v2.py
@@ -134,7 +134,7 @@ def _tokenize(self, text):
return self._tokenizer.tokenize(text)
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self._tokenizer.spm.PieceToId(token)
def _convert_id_to_token(self, index):
@@ -142,7 +142,7 @@ def _convert_id_to_token(self, index):
return self._tokenizer.spm.IdToPiece(index) if index < self.vocab_size else self.unk_token
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
return self._tokenizer.decode(tokens)
def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
@@ -428,7 +428,7 @@ def save_pretrained(self, path: str, filename_prefix: str = None):
def _is_whitespace(char):
"""Checks whether `chars` is a whitespace character."""
- # \t, \n, and \r are technically contorl characters but we treat them
+ # \t, \n, and \r are technically control characters but we treat them
# as whitespace since they are generally considered as such.
if char == " " or char == "\t" or char == "\n" or char == "\r":
return True
diff --git a/src/transformers/models/deit/modeling_deit.py b/src/transformers/models/deit/modeling_deit.py
index 8844d7f656fa..602d5e26005b 100644
--- a/src/transformers/models/deit/modeling_deit.py
+++ b/src/transformers/models/deit/modeling_deit.py
@@ -386,7 +386,7 @@ class DeiTPreTrainedModel(PreTrainedModel):
base_model_prefix = "deit"
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, (nn.Linear, nn.Conv2d)):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/distilbert/modeling_distilbert.py b/src/transformers/models/distilbert/modeling_distilbert.py
index 911fba808848..ca4b42987b41 100755
--- a/src/transformers/models/distilbert/modeling_distilbert.py
+++ b/src/transformers/models/distilbert/modeling_distilbert.py
@@ -167,11 +167,11 @@ def forward(self, query, key, value, mask, head_mask=None, output_attentions=Fal
mask_reshp = (bs, 1, 1, k_length)
def shape(x):
- """ separate heads """
+ """separate heads"""
return x.view(bs, -1, self.n_heads, dim_per_head).transpose(1, 2)
def unshape(x):
- """ group heads """
+ """group heads"""
return x.transpose(1, 2).contiguous().view(bs, -1, self.n_heads * dim_per_head)
q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head)
diff --git a/src/transformers/models/distilbert/modeling_tf_distilbert.py b/src/transformers/models/distilbert/modeling_tf_distilbert.py
index 8ec0060ab36f..2eddbffc1436 100644
--- a/src/transformers/models/distilbert/modeling_tf_distilbert.py
+++ b/src/transformers/models/distilbert/modeling_tf_distilbert.py
@@ -175,11 +175,11 @@ def call(self, query, key, value, mask, head_mask, output_attentions, training=F
mask_reshape = [bs, 1, 1, k_length]
def shape(x):
- """ separate heads """
+ """separate heads"""
return tf.transpose(tf.reshape(x, (bs, -1, self.n_heads, dim_per_head)), perm=(0, 2, 1, 3))
def unshape(x):
- """ group heads """
+ """group heads"""
return tf.reshape(tf.transpose(x, perm=(0, 2, 1, 3)), (bs, -1, self.n_heads * dim_per_head))
q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head)
diff --git a/src/transformers/models/electra/modeling_electra.py b/src/transformers/models/electra/modeling_electra.py
index 8f77289fe535..006a22f4c774 100644
--- a/src/transformers/models/electra/modeling_electra.py
+++ b/src/transformers/models/electra/modeling_electra.py
@@ -653,7 +653,7 @@ class ElectraPreTrainedModel(PreTrainedModel):
# Copied from transformers.models.bert.modeling_bert.BertPreTrainedModel._init_weights
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/flaubert/modeling_tf_flaubert.py b/src/transformers/models/flaubert/modeling_tf_flaubert.py
index da2f2d21c7a0..c6f43a4ced08 100644
--- a/src/transformers/models/flaubert/modeling_tf_flaubert.py
+++ b/src/transformers/models/flaubert/modeling_tf_flaubert.py
@@ -342,11 +342,11 @@ def call(self, input, mask, kv, cache, head_mask, output_attentions, training=Fa
mask_reshape = (bs, 1, qlen, klen) if len(shape_list(mask)) == 3 else (bs, 1, 1, klen)
def shape(x):
- """ projection """
+ """projection"""
return tf.transpose(tf.reshape(x, (bs, -1, self.n_heads, dim_per_head)), perm=(0, 2, 1, 3))
def unshape(x):
- """ compute context """
+ """compute context"""
return tf.reshape(tf.transpose(x, perm=(0, 2, 1, 3)), (bs, -1, self.n_heads * dim_per_head))
q = shape(self.q_lin(input)) # (bs, n_heads, qlen, dim_per_head)
diff --git a/src/transformers/models/fsmt/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py b/src/transformers/models/fsmt/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py
index b34eb36abfd1..2470492ac743 100755
--- a/src/transformers/models/fsmt/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py
+++ b/src/transformers/models/fsmt/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py
@@ -134,7 +134,7 @@ def convert_fsmt_checkpoint_to_pytorch(fsmt_checkpoint_path, pytorch_dump_folder
f.write(json.dumps(src_vocab, ensure_ascii=False, indent=json_indent))
# detect whether this is a do_lower_case situation, which can be derived by checking whether we
- # have at least one upcase letter in the source vocab
+ # have at least one uppercase letter in the source vocab
do_lower_case = True
for k in src_vocab.keys():
if not k.islower():
diff --git a/src/transformers/models/fsmt/modeling_fsmt.py b/src/transformers/models/fsmt/modeling_fsmt.py
index 8979cc7e549a..54da504ab8e0 100644
--- a/src/transformers/models/fsmt/modeling_fsmt.py
+++ b/src/transformers/models/fsmt/modeling_fsmt.py
@@ -240,7 +240,7 @@
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
FSMT uses the :obj:`eos_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -252,7 +252,7 @@
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`torch.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -486,7 +486,7 @@ def forward(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
Returns:
BaseModelOutput or Tuple comprised of:
@@ -696,7 +696,7 @@ def forward(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
cross_attn_head_mask (:obj:`torch.Tensor` of shape :obj:`(num_layers, num_heads)`, `optional`):
Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
diff --git a/src/transformers/models/fsmt/tokenization_fsmt.py b/src/transformers/models/fsmt/tokenization_fsmt.py
index 226d18cc3eab..ff99d75eeb77 100644
--- a/src/transformers/models/fsmt/tokenization_fsmt.py
+++ b/src/transformers/models/fsmt/tokenization_fsmt.py
@@ -374,7 +374,7 @@ def _tokenize(self, text, lang="en", bypass_tokenizer=False):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.encoder.get(token, self.encoder.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -382,7 +382,7 @@ def _convert_id_to_token(self, index):
return self.decoder.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
# remove BPE
tokens = [t.replace(" ", "").replace("", " ") for t in tokens]
diff --git a/src/transformers/models/funnel/modeling_funnel.py b/src/transformers/models/funnel/modeling_funnel.py
index 1f277498d124..1db4ab87f301 100644
--- a/src/transformers/models/funnel/modeling_funnel.py
+++ b/src/transformers/models/funnel/modeling_funnel.py
@@ -184,11 +184,11 @@ def __init__(self, config):
self.sin_dropout = nn.Dropout(config.hidden_dropout)
self.cos_dropout = nn.Dropout(config.hidden_dropout)
# Track where we are at in terms of pooling from the original input, e.g., by how much the sequence length was
- # dividide.
+ # divided.
self.pooling_mult = None
def init_attention_inputs(self, inputs_embeds, attention_mask=None, token_type_ids=None):
- """ Returns the attention inputs associated to the inputs of the model. """
+ """Returns the attention inputs associated to the inputs of the model."""
# inputs_embeds has shape batch_size x seq_len x d_model
# attention_mask and token_type_ids have shape batch_size x seq_len
self.pooling_mult = 1
@@ -218,7 +218,7 @@ def get_position_embeds(self, seq_len, dtype, device):
For the factorized attention, it returns the matrices (phi, pi, psi, omega) used in the paper, appendix A.2.2,
final formula.
- For the relative shif attention, it returns all possible vectors R used in the paper, appendix A.2.1, final
+ For the relative shift attention, it returns all possible vectors R used in the paper, appendix A.2.1, final
formula.
Paper link: https://arxiv.org/abs/2006.03236
@@ -383,7 +383,7 @@ def pool_tensor(self, tensor, mode="mean", stride=2):
return tensor
def pre_attention_pooling(self, output, attention_inputs):
- """ Pool `output` and the proper parts of `attention_inputs` before the attention layer. """
+ """Pool `output` and the proper parts of `attention_inputs` before the attention layer."""
position_embeds, token_type_mat, attention_mask, cls_mask = attention_inputs
if self.config.pool_q_only:
if self.config.attention_type == "factorized":
@@ -403,7 +403,7 @@ def pre_attention_pooling(self, output, attention_inputs):
return output, attention_inputs
def post_attention_pooling(self, attention_inputs):
- """ Pool the proper parts of `attention_inputs` after the attention layer. """
+ """Pool the proper parts of `attention_inputs` after the attention layer."""
position_embeds, token_type_mat, attention_mask, cls_mask = attention_inputs
if self.config.pool_q_only:
self.pooling_mult *= 2
@@ -457,7 +457,7 @@ def __init__(self, config, block_index):
self.scale = 1.0 / (d_head ** 0.5)
def relative_positional_attention(self, position_embeds, q_head, context_len, cls_mask=None):
- """ Relative attention score for the positional encodings """
+ """Relative attention score for the positional encodings"""
# q_head has shape batch_size x sea_len x n_head x d_head
if self.config.attention_type == "factorized":
# Notations from the paper, appending A.2.2, final formula (https://arxiv.org/abs/2006.03236)
@@ -499,7 +499,7 @@ def relative_positional_attention(self, position_embeds, q_head, context_len, cl
return positional_attn
def relative_token_type_attention(self, token_type_mat, q_head, cls_mask=None):
- """ Relative attention score for the token_type_ids """
+ """Relative attention score for the token_type_ids"""
if token_type_mat is None:
return 0
batch_size, seq_len, context_len = token_type_mat.shape
diff --git a/src/transformers/models/funnel/modeling_tf_funnel.py b/src/transformers/models/funnel/modeling_tf_funnel.py
index b4e53eafdf03..8c2541da0cee 100644
--- a/src/transformers/models/funnel/modeling_tf_funnel.py
+++ b/src/transformers/models/funnel/modeling_tf_funnel.py
@@ -139,7 +139,7 @@ def __init__(self, config):
self.pooling_mult = None
def init_attention_inputs(self, inputs_embeds, attention_mask=None, token_type_ids=None, training=False):
- """ Returns the attention inputs associated to the inputs of the model. """
+ """Returns the attention inputs associated to the inputs of the model."""
# inputs_embeds has shape batch_size x seq_len x d_model
# attention_mask and token_type_ids have shape batch_size x seq_len
self.pooling_mult = 1
@@ -169,7 +169,7 @@ def get_position_embeds(self, seq_len, training=False):
For the factorized attention, it returns the matrices (phi, pi, psi, omega) used in the paper, appendix A.2.2,
final formula.
- For the relative shif attention, it returns all possible vectors R used in the paper, appendix A.2.1, final
+ For the relative shift attention, it returns all possible vectors R used in the paper, appendix A.2.1, final
formula.
Paper link: https://arxiv.org/abs/2006.03236
@@ -328,7 +328,7 @@ def pool_tensor(self, tensor, mode="mean", stride=2):
return tf.squeeze(tensor, 2) if ndim == 2 else tensor
def pre_attention_pooling(self, output, attention_inputs):
- """ Pool `output` and the proper parts of `attention_inputs` before the attention layer. """
+ """Pool `output` and the proper parts of `attention_inputs` before the attention layer."""
position_embeds, token_type_mat, attention_mask, cls_mask = attention_inputs
if self.pool_q_only:
if self.attention_type == "factorized":
@@ -348,7 +348,7 @@ def pre_attention_pooling(self, output, attention_inputs):
return output, attention_inputs
def post_attention_pooling(self, attention_inputs):
- """ Pool the proper parts of `attention_inputs` after the attention layer. """
+ """Pool the proper parts of `attention_inputs` after the attention layer."""
position_embeds, token_type_mat, attention_mask, cls_mask = attention_inputs
if self.pool_q_only:
self.pooling_mult *= 2
@@ -424,7 +424,7 @@ def build(self, input_shape):
super().build(input_shape)
def relative_positional_attention(self, position_embeds, q_head, context_len, cls_mask=None):
- """ Relative attention score for the positional encodings """
+ """Relative attention score for the positional encodings"""
# q_head has shape batch_size x sea_len x n_head x d_head
if self.attention_type == "factorized":
# Notations from the paper, appending A.2.2, final formula (https://arxiv.org/abs/2006.03236)
@@ -470,7 +470,7 @@ def relative_positional_attention(self, position_embeds, q_head, context_len, cl
return positional_attn
def relative_token_type_attention(self, token_type_mat, q_head, cls_mask=None):
- """ Relative attention score for the token_type_ids """
+ """Relative attention score for the token_type_ids"""
if token_type_mat is None:
return 0
batch_size, seq_len, context_len = shape_list(token_type_mat)
@@ -723,7 +723,7 @@ def call(
@keras_serializable
class TFFunnelBaseLayer(tf.keras.layers.Layer):
- """ Base model without decoder """
+ """Base model without decoder"""
config_class = FunnelConfig
@@ -807,7 +807,7 @@ def call(
@keras_serializable
class TFFunnelMainLayer(tf.keras.layers.Layer):
- """ Base model with decoder """
+ """Base model with decoder"""
config_class = FunnelConfig
@@ -1009,7 +1009,7 @@ class TFFunnelForPreTrainingOutput(ModelOutput):
Args:
logits (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length)`):
Prediction scores of the head (scores for each token before SoftMax).
- hidden_states (:obj:`tuple(tf.ensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
+ hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
Tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of
shape :obj:`(batch_size, sequence_length, hidden_size)`.
diff --git a/src/transformers/models/gpt2/modeling_gpt2.py b/src/transformers/models/gpt2/modeling_gpt2.py
index babe8ac1aa04..22d009411087 100644
--- a/src/transformers/models/gpt2/modeling_gpt2.py
+++ b/src/transformers/models/gpt2/modeling_gpt2.py
@@ -1236,7 +1236,7 @@ def __init__(self, config):
@add_start_docstrings_to_model_forward(GPT2_INPUTS_DOCSTRING)
@add_code_sample_docstrings(
tokenizer_class=_TOKENIZER_FOR_DOC,
- checkpoint="microsoft/dialogrpt",
+ checkpoint="microsoft/DialogRPT-updown",
output_type=SequenceClassifierOutputWithPast,
config_class=_CONFIG_FOR_DOC,
)
diff --git a/src/transformers/models/gpt2/modeling_tf_gpt2.py b/src/transformers/models/gpt2/modeling_tf_gpt2.py
index cc7829871a05..32ee341814b5 100644
--- a/src/transformers/models/gpt2/modeling_tf_gpt2.py
+++ b/src/transformers/models/gpt2/modeling_tf_gpt2.py
@@ -70,7 +70,7 @@ def __init__(self, nx, n_ctx, config, scale=False, **kwargs):
super().__init__(**kwargs)
n_state = nx # in Attention: n_state=768 (nx=n_embd)
- # [switch nx => n_state from Block to Attention to keep identical to TF implem]
+ # [switch nx => n_state from Block to Attention to keep identical to TF implementation]
assert n_state % config.n_head == 0
self.n_ctx = n_ctx
self.n_head = config.n_head
diff --git a/src/transformers/models/gpt2/tokenization_gpt2.py b/src/transformers/models/gpt2/tokenization_gpt2.py
index e27ad9d3c003..d09e4eedd0e2 100644
--- a/src/transformers/models/gpt2/tokenization_gpt2.py
+++ b/src/transformers/models/gpt2/tokenization_gpt2.py
@@ -70,7 +70,7 @@ def bytes_to_unicode():
The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab
if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for
- decent coverage. This is a signficant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup
+ decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup
tables between utf-8 bytes and unicode strings.
"""
bs = (
@@ -189,7 +189,7 @@ def __init__(
self.cache = {}
self.add_prefix_space = add_prefix_space
- # Should haved added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions
+ # Should have added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions
self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")
@property
@@ -242,17 +242,17 @@ def bpe(self, token):
return word
def _tokenize(self, text):
- """ Tokenize a string. """
+ """Tokenize a string."""
bpe_tokens = []
for token in re.findall(self.pat, text):
token = "".join(
self.byte_encoder[b] for b in token.encode("utf-8")
- ) # Maps all our bytes to unicode strings, avoiding controle tokens of the BPE (spaces in our case)
+ ) # Maps all our bytes to unicode strings, avoiding control tokens of the BPE (spaces in our case)
bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(" "))
return bpe_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.encoder.get(token, self.encoder.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -260,7 +260,7 @@ def _convert_id_to_token(self, index):
return self.decoder.get(index)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
text = "".join(tokens)
text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
return text
diff --git a/src/transformers/models/gpt_neo/modeling_gpt_neo.py b/src/transformers/models/gpt_neo/modeling_gpt_neo.py
index e5d7c97c656a..ed4ad679360e 100755
--- a/src/transformers/models/gpt_neo/modeling_gpt_neo.py
+++ b/src/transformers/models/gpt_neo/modeling_gpt_neo.py
@@ -221,7 +221,7 @@ def create_local_attention_mask(batch_size, seq_length, window_size, device, att
if attention_mask is None:
attention_mask = torch.ones(batch_size, seq_length, dtype=torch.long, device=device)
- # A block can also be padded becuase of the _look_back operation
+ # A block can also be padded because of the _look_back operation
# look back into the attention_block such that it will also get padded the same way
# and have 0s in the padded position
attention_mask = GPTNeoAttentionMixin._look_back(attention_mask, block_length, window_size, is_key_value=False)
@@ -804,8 +804,8 @@ def forward(
# Prepare head mask if needed
# 1.0 in head_mask indicate we keep the head
- # attention_probs has shape bsz x num_headss x N x N
- # head_mask has shape n_layer x batch x num_headss x N x N
+ # attention_probs has shape bsz x num_heads x N x N
+ # head_mask has shape n_layer x batch x num_heads x N x N
head_mask = self.get_head_mask(head_mask, self.config.num_layers)
if inputs_embeds is None:
diff --git a/src/transformers/models/ibert/modeling_ibert.py b/src/transformers/models/ibert/modeling_ibert.py
index 382577a9f06c..3c72c2a17e27 100644
--- a/src/transformers/models/ibert/modeling_ibert.py
+++ b/src/transformers/models/ibert/modeling_ibert.py
@@ -645,7 +645,7 @@ class IBertPreTrainedModel(PreTrainedModel):
base_model_prefix = "ibert"
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, (QuantLinear, nn.Linear)):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/ibert/quant_modules.py b/src/transformers/models/ibert/quant_modules.py
index add4410ca619..065a3fef6144 100644
--- a/src/transformers/models/ibert/quant_modules.py
+++ b/src/transformers/models/ibert/quant_modules.py
@@ -35,11 +35,11 @@ class QuantEmbedding(nn.Module):
:obj:`torch.nn.Embedding`.
Args:
- weight_bit (:obj:`int`, `optiona`l, defaults to :obj:`8`):
+ weight_bit (:obj:`int`, `optional`, defaults to :obj:`8`):
Bitwidth for the quantized weight.
- momentum (:obj:`float`, `optional, defaults to :obj:`0.95`):
+ momentum (:obj:`float`, `optional`, defaults to :obj:`0.95`):
Momentum for updating the activation quantization range.
- quant_mode (:obj:`bool`, `optional, defaults to :obj:`False`):
+ quant_mode (:obj:`bool`, `optional`, defaults to :obj:`False`):
Whether or not the layer is quantized.
"""
@@ -619,7 +619,7 @@ def symmetric_linear_quantization_params(num_bits, saturation_min, saturation_ma
`saturation_max`.
"""
# in this part, we do not need any gradient computation,
- # in order to enfore this, we put torch.no_grad()
+ # in order to enforce this, we put torch.no_grad()
with torch.no_grad():
n = 2 ** (num_bits - 1) - 1
diff --git a/src/transformers/models/layoutlm/modeling_layoutlm.py b/src/transformers/models/layoutlm/modeling_layoutlm.py
index bce2ddd27534..c8c395557977 100644
--- a/src/transformers/models/layoutlm/modeling_layoutlm.py
+++ b/src/transformers/models/layoutlm/modeling_layoutlm.py
@@ -611,7 +611,7 @@ class LayoutLMPreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/led/modeling_led.py b/src/transformers/models/led/modeling_led.py
index b245c3250b50..7f9f91bc22f2 100755
--- a/src/transformers/models/led/modeling_led.py
+++ b/src/transformers/models/led/modeling_led.py
@@ -509,7 +509,7 @@ def _sliding_chunks_matmul_attn_probs_value(
@staticmethod
def _get_global_attn_indices(is_index_global_attn):
- """ compute global attn indices required throughout forward pass """
+ """compute global attn indices required throughout forward pass"""
# helper variable
num_global_attn_indices = is_index_global_attn.long().sum(dim=1)
@@ -842,9 +842,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -1499,7 +1499,7 @@ class LEDSeq2SeqQuestionAnsweringModelOutput(ModelOutput):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`torch.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -1689,7 +1689,7 @@ def forward(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
inputs_embeds (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
representation. This is useful if you want more control over how to convert :obj:`input_ids` indices
@@ -1920,7 +1920,7 @@ def forward(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
cross_attn_head_mask (:obj:`torch.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
diff --git a/src/transformers/models/led/modeling_tf_led.py b/src/transformers/models/led/modeling_tf_led.py
index 8197a8ad8056..7752044c22e5 100644
--- a/src/transformers/models/led/modeling_tf_led.py
+++ b/src/transformers/models/led/modeling_tf_led.py
@@ -670,7 +670,7 @@ def _chunk(hidden_states, window_overlap):
@staticmethod
def _get_global_attn_indices(is_index_global_attn):
- """ compute global attn indices required throughout forward pass """
+ """compute global attn indices required throughout forward pass"""
# helper variable
num_global_attn_indices = tf.math.count_nonzero(is_index_global_attn, axis=1)
num_global_attn_indices = tf.cast(num_global_attn_indices, dtype=tf.constant(1).dtype)
@@ -869,7 +869,7 @@ def _compute_global_attn_output_from_hidden(
# compute global attn probs
global_attn_probs_float = tf.nn.softmax(global_attn_scores, axis=-1)
- # apply layer head maskin
+ # apply layer head masking
if layer_head_mask is not None:
if tf.executing_eagerly():
tf.debugging.assert_equal(
@@ -1552,7 +1552,7 @@ class TFLEDSeq2SeqLMOutput(ModelOutput):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -1667,7 +1667,7 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
@@ -1926,14 +1926,14 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
encoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention
on hidden heads. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
past_key_values (:obj:`Tuple[Tuple[tf.Tensor]]` of length :obj:`config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up
diff --git a/src/transformers/models/longformer/modeling_longformer.py b/src/transformers/models/longformer/modeling_longformer.py
index 65634ca314d3..9ca1d9a182e7 100755
--- a/src/transformers/models/longformer/modeling_longformer.py
+++ b/src/transformers/models/longformer/modeling_longformer.py
@@ -393,7 +393,7 @@ class LongformerTokenClassifierOutput(ModelOutput):
def _get_question_end_index(input_ids, sep_token_id):
"""
- Computes the index of the first occurance of `sep_token_id`.
+ Computes the index of the first occurrence of `sep_token_id`.
"""
sep_token_indices = (input_ids == sep_token_id).nonzero()
@@ -899,7 +899,7 @@ def _sliding_chunks_matmul_attn_probs_value(
@staticmethod
def _get_global_attn_indices(is_index_global_attn):
- """ compute global attn indices required throughout forward pass """
+ """compute global attn indices required throughout forward pass"""
# helper variable
num_global_attn_indices = is_index_global_attn.long().sum(dim=1)
@@ -1363,7 +1363,7 @@ class LongformerPreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
@@ -1428,7 +1428,7 @@ def _init_weights(self, module):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`torch.Tensor` of shape :obj:`(num_layers, num_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
diff --git a/src/transformers/models/longformer/modeling_tf_longformer.py b/src/transformers/models/longformer/modeling_tf_longformer.py
index 6d5f7692834f..bb193fbf6389 100644
--- a/src/transformers/models/longformer/modeling_tf_longformer.py
+++ b/src/transformers/models/longformer/modeling_tf_longformer.py
@@ -1189,7 +1189,7 @@ def _chunk(hidden_states, window_overlap):
@staticmethod
def _get_global_attn_indices(is_index_global_attn):
- """ compute global attn indices required throughout forward pass """
+ """compute global attn indices required throughout forward pass"""
# helper variable
num_global_attn_indices = tf.math.count_nonzero(is_index_global_attn, axis=1)
num_global_attn_indices = tf.cast(num_global_attn_indices, dtype=tf.constant(1).dtype)
@@ -1388,7 +1388,7 @@ def _compute_global_attn_output_from_hidden(
# compute global attn probs
global_attn_probs_float = tf.nn.softmax(global_attn_scores, axis=-1)
- # apply layer head maskin
+ # apply layer head masking
if layer_head_mask is not None:
if tf.executing_eagerly():
tf.debugging.assert_equal(
@@ -1707,7 +1707,7 @@ def call(
inputs["attention_mask"], (attention_mask_shape[0], attention_mask_shape[1], 1, 1)
)
- # Since attention_mask is 1.0 for positions we want to locall attend locally and 0.0 for
+ # Since attention_mask is 1.0 for positions we want to attend locally and 0.0 for
# masked and global attn positions, this operation will create a tensor which is 0.0 for
# positions we want to attend and -10000.0 for masked positions.
# Since we are adding it to the raw scores before the softmax, this is
@@ -1920,7 +1920,7 @@ def serving(self, inputs):
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
global_attention_mask (:obj:`tf.Tensor` of shape :obj:`({0})`, `optional`):
Mask to decide the attention given on each token, local attention or global attention. Tokens with global
diff --git a/src/transformers/models/lxmert/configuration_lxmert.py b/src/transformers/models/lxmert/configuration_lxmert.py
index 8c3ca17187a5..85f191de6b11 100644
--- a/src/transformers/models/lxmert/configuration_lxmert.py
+++ b/src/transformers/models/lxmert/configuration_lxmert.py
@@ -95,9 +95,9 @@ class LxmertConfig(PretrainedConfig):
Whether or not to add masked language modeling (as used in pretraining models such as BERT) to the loss
objective.
task_obj_predict (:obj:`bool`, `optional`, defaults to :obj:`True`):
- Whether or not to add object prediction, attribute ppredictionand feature regression to the loss objective.
+ Whether or not to add object prediction, attribute prediction and feature regression to the loss objective.
task_qa (:obj:`bool`, `optional`, defaults to :obj:`True`):
- Whether or not to add the question-asansweringoss to the objective
+ Whether or not to add the question-answering loss to the objective
visual_obj_loss (:obj:`bool`, `optional`, defaults to :obj:`True`):
Whether or not to calculate the object-prediction loss objective
visual_attr_loss (:obj:`bool`, `optional`, defaults to :obj:`True`):
diff --git a/src/transformers/models/lxmert/modeling_lxmert.py b/src/transformers/models/lxmert/modeling_lxmert.py
index 7610d5c0c523..cc7c22fe9b10 100644
--- a/src/transformers/models/lxmert/modeling_lxmert.py
+++ b/src/transformers/models/lxmert/modeling_lxmert.py
@@ -783,7 +783,7 @@ class LxmertPreTrainedModel(PreTrainedModel):
base_model_prefix = "lxmert"
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/m2m_100/modeling_m2m_100.py b/src/transformers/models/m2m_100/modeling_m2m_100.py
index 7fccb32c4d43..5d01e0912982 100755
--- a/src/transformers/models/m2m_100/modeling_m2m_100.py
+++ b/src/transformers/models/m2m_100/modeling_m2m_100.py
@@ -306,9 +306,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -595,7 +595,7 @@ def _init_weights(self, module):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
M2M100 uses the :obj:`eos_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
diff --git a/src/transformers/models/marian/modeling_marian.py b/src/transformers/models/marian/modeling_marian.py
index b393697e6218..c99d4aa83249 100755
--- a/src/transformers/models/marian/modeling_marian.py
+++ b/src/transformers/models/marian/modeling_marian.py
@@ -252,9 +252,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -559,7 +559,7 @@ def dummy_inputs(self):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
Marian uses the :obj:`pad_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -1335,7 +1335,7 @@ def prepare_inputs_for_generation(
def prepare_decoder_input_ids_from_labels(self, labels: torch.Tensor):
return shift_tokens_right(labels, self.config.pad_token_id, self.config.decoder_start_token_id)
- def adjust_logits_during_generation(self, logits, cur_len, max_length):
+ def adjust_logits_during_generation(self, logits, cur_len):
logits[:, self.config.pad_token_id] = float("-inf") # never predict pad token.
return logits
diff --git a/src/transformers/models/marian/modeling_tf_marian.py b/src/transformers/models/marian/modeling_tf_marian.py
index 15271f8b22bd..81ad6b81850d 100644
--- a/src/transformers/models/marian/modeling_tf_marian.py
+++ b/src/transformers/models/marian/modeling_tf_marian.py
@@ -31,7 +31,7 @@
)
from ...modeling_tf_outputs import (
TFBaseModelOutput,
- TFBaseModelOutputWithPast,
+ TFBaseModelOutputWithPastAndCrossAttentions,
TFSeq2SeqLMOutput,
TFSeq2SeqModelOutput,
)
@@ -408,7 +408,7 @@ def call(
encoder_hidden_states: Optional[tf.Tensor] = None,
encoder_attention_mask: Optional[tf.Tensor] = None,
layer_head_mask: Optional[tf.Tensor] = None,
- encoder_layer_head_mask: Optional[tf.Tensor] = None,
+ cross_attn_layer_head_mask: Optional[tf.Tensor] = None,
past_key_value: Optional[Tuple[tf.Tensor]] = None,
training=False,
) -> Tuple[tf.Tensor, tf.Tensor, Tuple[Tuple[tf.Tensor]]]:
@@ -422,8 +422,8 @@ def call(
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
layer_head_mask (:obj:`tf.Tensor`): mask for attention heads in a given layer of size
`(decoder_attention_heads,)`
- encoder_layer_head_mask (:obj:`tf.Tensor`): mask for encoder attention heads in a given layer of size
- `(encoder_attention_heads,)`
+ cross_attn_layer_head_mask (:obj:`tf.Tensor`): mask for heads of the cross-attention module.
+ `(decoder_attention_heads,)`
past_key_value (:obj:`Tuple(tf.Tensor)`): cached past key and value projection states
"""
residual = hidden_states
@@ -444,16 +444,17 @@ def call(
# Cross-Attention Block
cross_attn_present_key_value = None
+ cross_attn_weights = None
if encoder_hidden_states is not None:
residual = hidden_states
# cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
- hidden_states, _, cross_attn_present_key_value = self.encoder_attn(
+ hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
hidden_states=hidden_states,
key_value_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
- layer_head_mask=encoder_layer_head_mask,
+ layer_head_mask=cross_attn_layer_head_mask,
past_key_value=cross_attn_past_key_value,
)
hidden_states = self.dropout(hidden_states, training=training)
@@ -475,6 +476,7 @@ def call(
return (
hidden_states,
self_attn_weights,
+ cross_attn_weights,
present_key_value,
)
@@ -592,7 +594,7 @@ def serving(self, inputs):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
Marian uses the :obj:`pad_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -603,7 +605,7 @@ def serving(self, inputs):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -611,6 +613,12 @@ def serving(self, inputs):
- 1 indicates the head is **not masked**,
- 0 indicates the head is **masked**.
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
encoder_outputs (:obj:`tf.FloatTensor`, `optional`):
hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
of shape :obj:`(batch_size, sequence_length, hidden_size)` is a sequence of
@@ -707,7 +715,7 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
@@ -848,7 +856,7 @@ def call(
encoder_hidden_states=None,
encoder_attention_mask=None,
head_mask=None,
- encoder_head_mask=None,
+ cross_attn_head_mask=None,
past_key_values=None,
use_cache=None,
output_attentions=None,
@@ -890,14 +898,13 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
- encoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional`):
- Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention
- on hidden heads. Mask values selected in ``[0, 1]``:
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
past_key_values (:obj:`Tuple[Tuple[tf.Tensor]]` of length :obj:`config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up
@@ -934,7 +941,7 @@ def call(
encoder_hidden_states=encoder_hidden_states,
encoder_attention_mask=encoder_attention_mask,
head_mask=head_mask,
- encoder_head_mask=encoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
inputs_embeds=inputs_embeds,
past_key_values=past_key_values,
use_cache=use_cache,
@@ -986,19 +993,21 @@ def call(
hidden_states = self.dropout(hidden_states + positions, training=inputs["training"])
# decoder layers
- all_hidden_states = ()
- all_self_attns = ()
- present_key_values = ()
+ all_hidden_states = () if inputs["output_hidden_states"] else None
+ all_self_attns = () if inputs["output_attentions"] else None
+ all_cross_attns = () if (inputs["output_attentions"] and inputs["encoder_hidden_states"] is not None) else None
+ present_key_values = () if inputs["use_cache"] else None
- # check if head_mask has a correct number of layers specified if desired
+ # check if head_mask and cross_attn_head_mask have a correct number of layers specified if desired
# The tf.debugging asserts are not compliant with XLA then they
# have to be disabled in other modes than eager.
- if inputs["head_mask"] is not None and tf.executing_eagerly():
- tf.debugging.assert_equal(
- shape_list(inputs["head_mask"])[0],
- len(self.layers),
- message=f"The head_mask should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs['head_mask'])[0]}.",
- )
+ for attn_mask in ["head_mask", "cross_attn_head_mask"]:
+ if inputs[attn_mask] is not None and tf.executing_eagerly():
+ tf.debugging.assert_equal(
+ shape_list(inputs[attn_mask])[0],
+ len(self.layers),
+ message=f"The {attn_mask} should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs[attn_mask])[0]}.",
+ )
for idx, decoder_layer in enumerate(self.layers):
# add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
@@ -1011,14 +1020,14 @@ def call(
past_key_value = inputs["past_key_values"][idx] if inputs["past_key_values"] is not None else None
- hidden_states, layer_self_attn, present_key_value = decoder_layer(
+ hidden_states, layer_self_attn, layer_cross_attn, present_key_value = decoder_layer(
hidden_states,
attention_mask=combined_attention_mask,
encoder_hidden_states=inputs["encoder_hidden_states"],
encoder_attention_mask=inputs["encoder_attention_mask"],
layer_head_mask=inputs["head_mask"][idx] if inputs["head_mask"] is not None else None,
- encoder_layer_head_mask=inputs["encoder_head_mask"][idx]
- if inputs["encoder_head_mask"] is not None
+ cross_attn_layer_head_mask=inputs["cross_attn_head_mask"][idx]
+ if inputs["cross_attn_head_mask"] is not None
else None,
past_key_value=past_key_value,
)
@@ -1029,23 +1038,30 @@ def call(
if inputs["output_attentions"]:
all_self_attns += (layer_self_attn,)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns += (layer_cross_attn,)
+
if inputs["output_hidden_states"]:
all_hidden_states += (hidden_states,)
- else:
- all_hidden_states = None
- all_self_attns = list(all_self_attns) if inputs["output_attentions"] else None
+ if inputs["output_attentions"]:
+ all_self_attns = list(all_self_attns)
+
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns = list(all_cross_attns)
- present_key_values = (encoder_hidden_states, present_key_values) if inputs["use_cache"] else None
+ if inputs["use_cache"]:
+ present_key_values = (inputs["encoder_hidden_states"], present_key_values)
if not inputs["return_dict"]:
- return hidden_states, present_key_values, all_hidden_states, all_self_attns
+ return hidden_states, present_key_values, all_hidden_states, all_self_attns, all_cross_attns
else:
- return TFBaseModelOutputWithPast(
+ return TFBaseModelOutputWithPastAndCrossAttentions(
last_hidden_state=hidden_states,
past_key_values=present_key_values,
hidden_states=all_hidden_states,
attentions=all_self_attns,
+ cross_attentions=all_cross_attns,
)
@@ -1092,6 +1108,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1112,6 +1129,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1161,7 +1179,7 @@ def call(
encoder_hidden_states=inputs["encoder_outputs"][0],
encoder_attention_mask=inputs["attention_mask"],
head_mask=inputs["decoder_head_mask"],
- encoder_head_mask=inputs["head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["decoder_inputs_embeds"],
use_cache=inputs["use_cache"],
@@ -1179,6 +1197,7 @@ def call(
past_key_values=decoder_outputs.past_key_values,
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
+ cross_attentions=decoder_outputs.cross_attentions,
encoder_last_hidden_state=inputs["encoder_outputs"].last_hidden_state,
encoder_hidden_states=inputs["encoder_outputs"].hidden_states,
encoder_attentions=inputs["encoder_outputs"].attentions,
@@ -1216,6 +1235,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1235,6 +1255,7 @@ def call(
decoder_input_ids=decoder_input_ids,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
decoder_attention_mask=decoder_attention_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
@@ -1255,6 +1276,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
encoder_outputs=inputs["encoder_outputs"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
@@ -1273,6 +1295,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1281,6 +1304,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
@@ -1335,6 +1359,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[TFBaseModelOutput] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1365,6 +1390,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1398,6 +1424,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
decoder_inputs_embeds=inputs["decoder_inputs_embeds"],
@@ -1420,6 +1447,7 @@ def call(
past_key_values=outputs.past_key_values, # index 1 of d outputs
decoder_hidden_states=outputs.decoder_hidden_states, # index 2 of d outputs
decoder_attentions=outputs.decoder_attentions, # index 3 of d outputs
+ cross_attentions=outputs.cross_attentions, # index 4 of d outputs
encoder_last_hidden_state=outputs.last_hidden_state, # index 0 of encoder outputs
encoder_hidden_states=outputs.encoder_hidden_states, # 1 of e out
encoder_attentions=outputs.encoder_attentions, # 2 of e out
@@ -1430,6 +1458,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1438,6 +1467,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
diff --git a/src/transformers/models/marian/tokenization_marian.py b/src/transformers/models/marian/tokenization_marian.py
index 613b385b7799..13453f0b58c8 100644
--- a/src/transformers/models/marian/tokenization_marian.py
+++ b/src/transformers/models/marian/tokenization_marian.py
@@ -227,7 +227,7 @@ def decode(self, token_ids, **kwargs):
return super().decode(token_ids, **kwargs)
def convert_tokens_to_string(self, tokens: List[str]) -> str:
- """Uses source spm if _decode_use_source_tokenizer is True, and target spm otherwise """
+ """Uses source spm if _decode_use_source_tokenizer is True, and target spm otherwise"""
if self._decode_use_source_tokenizer:
return self.spm_source.DecodePieces(tokens)
else:
diff --git a/src/transformers/models/mbart/modeling_mbart.py b/src/transformers/models/mbart/modeling_mbart.py
index e9719795d37c..dd76e6512902 100755
--- a/src/transformers/models/mbart/modeling_mbart.py
+++ b/src/transformers/models/mbart/modeling_mbart.py
@@ -117,7 +117,7 @@ class MBartLearnedPositionalEmbedding(nn.Embedding):
def __init__(self, num_embeddings: int, embedding_dim: int):
# MBart is set up so that if padding_idx is specified then offset the embedding ids by 2
- # and adjust num_embeddings appropriately. Other models dont have this hack
+ # and adjust num_embeddings appropriately. Other models don't have this hack
self.offset = 2
super().__init__(num_embeddings + self.offset, embedding_dim)
@@ -243,9 +243,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -582,7 +582,7 @@ def dummy_inputs(self):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
MBart uses a specific language id token as the starting token for :obj:`decoder_input_ids` generation that
varies according to source and target language, *e.g.* 25004 for `en_XX`, and 25003 for `de_DE`. If
diff --git a/src/transformers/models/mbart/modeling_tf_mbart.py b/src/transformers/models/mbart/modeling_tf_mbart.py
index ea3294aa5a5d..a17d9ad1a0a6 100644
--- a/src/transformers/models/mbart/modeling_tf_mbart.py
+++ b/src/transformers/models/mbart/modeling_tf_mbart.py
@@ -30,7 +30,7 @@
)
from ...modeling_tf_outputs import (
TFBaseModelOutput,
- TFBaseModelOutputWithPast,
+ TFBaseModelOutputWithPastAndCrossAttentions,
TFSeq2SeqLMOutput,
TFSeq2SeqModelOutput,
)
@@ -118,7 +118,7 @@ class TFMBartLearnedPositionalEmbedding(TFSharedEmbeddings):
def __init__(self, num_embeddings: int, embedding_dim: int, **kwargs):
# MBart is set up so that if padding_idx is specified then offset the embedding ids by 2
- # and adjust num_embeddings appropriately. Other models dont have this hack
+ # and adjust num_embeddings appropriately. Other models don't have this hack
self.offset = 2
super().__init__(num_embeddings + self.offset, embedding_dim, **kwargs)
@@ -368,7 +368,7 @@ def call(
encoder_hidden_states: Optional[tf.Tensor] = None,
encoder_attention_mask: Optional[tf.Tensor] = None,
layer_head_mask: Optional[tf.Tensor] = None,
- encoder_layer_head_mask: Optional[tf.Tensor] = None,
+ cross_attn_layer_head_mask: Optional[tf.Tensor] = None,
past_key_value: Optional[Tuple[tf.Tensor]] = None,
training=False,
) -> Tuple[tf.Tensor, tf.Tensor, Tuple[Tuple[tf.Tensor]]]:
@@ -382,8 +382,8 @@ def call(
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
layer_head_mask (:obj:`tf.Tensor`): mask for attention heads in a given layer of size
`(decoder_attention_heads,)`
- encoder_layer_head_mask (:obj:`tf.Tensor`): mask for encoder attention heads in a given layer of size
- `(encoder_attention_heads,)`
+ cross_attn_layer_head_mask (:obj:`tf.Tensor`): mask for heads of the cross-attention module.
+ `(decoder_attention_heads,)`
past_key_value (:obj:`Tuple(tf.Tensor)`): cached past key and value projection states
"""
residual = hidden_states
@@ -404,17 +404,18 @@ def call(
# Cross-Attention Block
cross_attn_present_key_value = None
+ cross_attn_weights = None
if encoder_hidden_states is not None:
residual = hidden_states
hidden_states = self.encoder_attn_layer_norm(hidden_states)
# cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
- hidden_states, _, cross_attn_present_key_value = self.encoder_attn(
+ hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
hidden_states=hidden_states,
key_value_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
- layer_head_mask=encoder_layer_head_mask,
+ layer_head_mask=cross_attn_layer_head_mask,
past_key_value=cross_attn_past_key_value,
)
hidden_states = self.dropout(hidden_states, training=training)
@@ -435,6 +436,7 @@ def call(
return (
hidden_states,
self_attn_weights,
+ cross_attn_weights,
present_key_value,
)
@@ -531,7 +533,7 @@ def serving(self, inputs):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
MBart uses a specific language id token as the starting token for :obj:`decoder_input_ids` generation that
varies according to source and target language, *e.g.* 25004 for `en_XX`, and 25003 for `de_DE`. If
@@ -547,7 +549,7 @@ def serving(self, inputs):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -555,6 +557,12 @@ def serving(self, inputs):
- 1 indicates the head is **not masked**,
- 0 indicates the head is **masked**.
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
encoder_outputs (:obj:`tf.FloatTensor`, `optional`):
hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
of shape :obj:`(batch_size, sequence_length, hidden_size)` is a sequence of
@@ -682,7 +690,7 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
@@ -828,7 +836,7 @@ def call(
encoder_hidden_states=None,
encoder_attention_mask=None,
head_mask=None,
- encoder_head_mask=None,
+ cross_attn_head_mask=None,
past_key_values=None,
use_cache=None,
output_attentions=None,
@@ -870,14 +878,13 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
- encoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional`):
- Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention
- on hidden heads. Mask values selected in ``[0, 1]``:
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
past_key_values (:obj:`Tuple[Tuple[tf.Tensor]]` of length :obj:`config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up
@@ -914,7 +921,7 @@ def call(
encoder_hidden_states=encoder_hidden_states,
encoder_attention_mask=encoder_attention_mask,
head_mask=head_mask,
- encoder_head_mask=encoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
inputs_embeds=inputs_embeds,
past_key_values=past_key_values,
use_cache=use_cache,
@@ -967,19 +974,21 @@ def call(
hidden_states = self.dropout(hidden_states, training=inputs["training"])
# decoder layers
- all_hidden_states = ()
- all_self_attns = ()
- present_key_values = ()
+ all_hidden_states = () if inputs["output_hidden_states"] else None
+ all_self_attns = () if inputs["output_attentions"] else None
+ all_cross_attns = () if (inputs["output_attentions"] and inputs["encoder_hidden_states"] is not None) else None
+ present_key_values = () if inputs["use_cache"] else None
- # check if head_mask has a correct number of layers specified if desired
+ # check if head_mask and cross_attn_head_mask have a correct number of layers specified if desired
# The tf.debugging asserts are not compliant with XLA then they
# have to be disabled in other modes than eager.
- if inputs["head_mask"] is not None and tf.executing_eagerly():
- tf.debugging.assert_equal(
- shape_list(inputs["head_mask"])[0],
- len(self.layers),
- message=f"The head_mask should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs['head_mask'])[0]}.",
- )
+ for attn_mask in ["head_mask", "cross_attn_head_mask"]:
+ if inputs[attn_mask] is not None and tf.executing_eagerly():
+ tf.debugging.assert_equal(
+ shape_list(inputs[attn_mask])[0],
+ len(self.layers),
+ message=f"The {attn_mask} should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs[attn_mask])[0]}.",
+ )
for idx, decoder_layer in enumerate(self.layers):
# add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
@@ -992,14 +1001,14 @@ def call(
past_key_value = inputs["past_key_values"][idx] if inputs["past_key_values"] is not None else None
- hidden_states, layer_self_attn, present_key_value = decoder_layer(
+ hidden_states, layer_self_attn, layer_cross_attn, present_key_value = decoder_layer(
hidden_states,
attention_mask=combined_attention_mask,
encoder_hidden_states=inputs["encoder_hidden_states"],
encoder_attention_mask=inputs["encoder_attention_mask"],
layer_head_mask=inputs["head_mask"][idx] if inputs["head_mask"] is not None else None,
- encoder_layer_head_mask=inputs["encoder_head_mask"][idx]
- if inputs["encoder_head_mask"] is not None
+ cross_attn_layer_head_mask=inputs["cross_attn_head_mask"][idx]
+ if inputs["cross_attn_head_mask"] is not None
else None,
past_key_value=past_key_value,
)
@@ -1010,25 +1019,32 @@ def call(
if inputs["output_attentions"]:
all_self_attns += (layer_self_attn,)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns += (layer_cross_attn,)
+
hidden_states = self.layer_norm(hidden_states)
if inputs["output_hidden_states"]:
all_hidden_states += (hidden_states,)
- else:
- all_hidden_states = None
- all_self_attns = list(all_self_attns) if inputs["output_attentions"] else None
+ if inputs["output_attentions"]:
+ all_self_attns = list(all_self_attns)
+
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns = list(all_cross_attns)
- present_key_values = (encoder_hidden_states, present_key_values) if inputs["use_cache"] else None
+ if inputs["use_cache"]:
+ present_key_values = (inputs["encoder_hidden_states"], present_key_values)
if not inputs["return_dict"]:
- return hidden_states, present_key_values, all_hidden_states, all_self_attns
+ return hidden_states, present_key_values, all_hidden_states, all_self_attns, all_cross_attns
else:
- return TFBaseModelOutputWithPast(
+ return TFBaseModelOutputWithPastAndCrossAttentions(
last_hidden_state=hidden_states,
past_key_values=present_key_values,
hidden_states=all_hidden_states,
attentions=all_self_attns,
+ cross_attentions=all_cross_attns,
)
@@ -1075,6 +1091,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1095,6 +1112,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1147,7 +1165,7 @@ def call(
encoder_hidden_states=inputs["encoder_outputs"][0],
encoder_attention_mask=inputs["attention_mask"],
head_mask=inputs["decoder_head_mask"],
- encoder_head_mask=inputs["head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["decoder_inputs_embeds"],
use_cache=inputs["use_cache"],
@@ -1165,6 +1183,7 @@ def call(
past_key_values=decoder_outputs.past_key_values,
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
+ cross_attentions=decoder_outputs.cross_attentions,
encoder_last_hidden_state=inputs["encoder_outputs"].last_hidden_state,
encoder_hidden_states=inputs["encoder_outputs"].hidden_states,
encoder_attentions=inputs["encoder_outputs"].attentions,
@@ -1202,6 +1221,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1222,6 +1242,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1241,6 +1262,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
encoder_outputs=inputs["encoder_outputs"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
@@ -1259,6 +1281,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1267,6 +1290,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
@@ -1321,6 +1345,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[TFBaseModelOutput] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1351,6 +1376,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1382,6 +1408,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
decoder_inputs_embeds=inputs["decoder_inputs_embeds"],
@@ -1404,6 +1431,7 @@ def call(
past_key_values=outputs.past_key_values, # index 1 of d outputs
decoder_hidden_states=outputs.decoder_hidden_states, # index 2 of d outputs
decoder_attentions=outputs.decoder_attentions, # index 3 of d outputs
+ cross_attentions=outputs.cross_attentions, # index 4 of d outputs
encoder_last_hidden_state=outputs.encoder_last_hidden_state, # index 0 of encoder outputs
encoder_hidden_states=outputs.encoder_hidden_states, # 1 of e out
encoder_attentions=outputs.encoder_attentions, # 2 of e out
@@ -1414,6 +1442,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1422,6 +1451,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
diff --git a/src/transformers/models/mbart/tokenization_mbart50.py b/src/transformers/models/mbart/tokenization_mbart50.py
index 48fdfe7772dd..ef7ec88f2446 100644
--- a/src/transformers/models/mbart/tokenization_mbart50.py
+++ b/src/transformers/models/mbart/tokenization_mbart50.py
@@ -189,7 +189,7 @@ def _tokenize(self, text: str) -> List[str]:
return self.sp_model.EncodeAsPieces(text)
def _convert_token_to_id(self, token: str) -> int:
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
if token in self.fairseq_tokens_to_ids:
return self.fairseq_tokens_to_ids[token]
spm_id = self.sp_model.PieceToId(token)
diff --git a/src/transformers/models/megatron_bert/modeling_megatron_bert.py b/src/transformers/models/megatron_bert/modeling_megatron_bert.py
index ce4ece3d32fb..49969c06b8f7 100755
--- a/src/transformers/models/megatron_bert/modeling_megatron_bert.py
+++ b/src/transformers/models/megatron_bert/modeling_megatron_bert.py
@@ -708,7 +708,7 @@ class MegatronBertPreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, (nn.Linear, nn.Embedding)):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/mobilebert/modeling_mobilebert.py b/src/transformers/models/mobilebert/modeling_mobilebert.py
index bd3f86d21e12..53c721a306e4 100644
--- a/src/transformers/models/mobilebert/modeling_mobilebert.py
+++ b/src/transformers/models/mobilebert/modeling_mobilebert.py
@@ -669,7 +669,7 @@ class MobileBertPreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
@@ -1487,7 +1487,7 @@ def forward(
@add_start_docstrings(
"""
- MoibleBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g.
+ MobileBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g.
for Named-Entity-Recognition (NER) tasks.
""",
MOBILEBERT_START_DOCSTRING,
diff --git a/src/transformers/models/mpnet/modeling_mpnet.py b/src/transformers/models/mpnet/modeling_mpnet.py
index e64d4de30b25..f1327a871976 100644
--- a/src/transformers/models/mpnet/modeling_mpnet.py
+++ b/src/transformers/models/mpnet/modeling_mpnet.py
@@ -56,7 +56,7 @@ class MPNetPreTrainedModel(PreTrainedModel):
base_model_prefix = "mpnet"
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/mpnet/modeling_tf_mpnet.py b/src/transformers/models/mpnet/modeling_tf_mpnet.py
index b9362bd6252d..dff6324e6c0e 100644
--- a/src/transformers/models/mpnet/modeling_tf_mpnet.py
+++ b/src/transformers/models/mpnet/modeling_tf_mpnet.py
@@ -430,7 +430,7 @@ def _relative_position_bucket(relative_position, num_buckets=32, max_distance=12
return ret
def compute_position_bias(self, x, position_ids=None):
- """ Compute binned relative position bias """
+ """Compute binned relative position bias"""
input_shape = shape_list(x)
qlen, klen = input_shape[1], input_shape[1]
@@ -674,7 +674,7 @@ def call(
@add_start_docstrings(
- "The bare MPNet Model transformer outputing raw hidden-states without any specific head on top.",
+ "The bare MPNet Model transformer outputting raw hidden-states without any specific head on top.",
MPNET_START_DOCSTRING,
)
class TFMPNetModel(TFMPNetPreTrainedModel):
diff --git a/src/transformers/models/mpnet/tokenization_mpnet.py b/src/transformers/models/mpnet/tokenization_mpnet.py
index 8041ec4ec5f7..98af763ade64 100644
--- a/src/transformers/models/mpnet/tokenization_mpnet.py
+++ b/src/transformers/models/mpnet/tokenization_mpnet.py
@@ -210,7 +210,7 @@ def _tokenize(self, text):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.vocab.get(token, self.vocab.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -218,7 +218,7 @@ def _convert_id_to_token(self, index):
return self.ids_to_tokens.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = " ".join(tokens).replace(" ##", "").strip()
return out_string
diff --git a/src/transformers/models/mpnet/tokenization_mpnet_fast.py b/src/transformers/models/mpnet/tokenization_mpnet_fast.py
index 07547fce57a4..0c426e7a41b3 100644
--- a/src/transformers/models/mpnet/tokenization_mpnet_fast.py
+++ b/src/transformers/models/mpnet/tokenization_mpnet_fast.py
@@ -154,7 +154,7 @@ def mask_token(self) -> str:
:obj:`str`: Mask token, to use when training a model with masked-language modeling. Log an error if used while
not having been set.
- MPNet tokenizer has a special mask token to be usble in the fill-mask pipeline. The mask token will greedily
+ MPNet tokenizer has a special mask token to be usable in the fill-mask pipeline. The mask token will greedily
comprise the space before the ``.
"""
if self._mask_token is None and self.verbose:
diff --git a/src/transformers/models/openai/modeling_openai.py b/src/transformers/models/openai/modeling_openai.py
index 6564a8fa42cf..27d5ef697d97 100644
--- a/src/transformers/models/openai/modeling_openai.py
+++ b/src/transformers/models/openai/modeling_openai.py
@@ -146,7 +146,7 @@ class Attention(nn.Module):
def __init__(self, nx, n_ctx, config, scale=False):
super().__init__()
n_state = nx # in Attention: n_state=768 (nx=n_embd)
- # [switch nx => n_state from Block to Attention to keep identical to TF implem]
+ # [switch nx => n_state from Block to Attention to keep identical to TF implementation]
assert n_state % config.n_head == 0
self.register_buffer("bias", torch.tril(torch.ones(n_ctx, n_ctx)).view(1, 1, n_ctx, n_ctx))
self.n_head = config.n_head
@@ -178,7 +178,7 @@ def _attn(self, q, k, v, attention_mask=None, head_mask=None, output_attentions=
w = torch.matmul(q, k)
if self.scale:
w = w / math.sqrt(v.size(-1))
- # w = w * self.bias + -1e9 * (1 - self.bias) # TF implem method: mask_attn_weights
+ # w = w * self.bias + -1e9 * (1 - self.bias) # TF implementation method: mask_attn_weights
# XD: self.b may be larger than w, so we need to crop it
b = self.bias[:, :, : w.size(-2), : w.size(-1)]
w = w * b + -1e4 * (1 - b)
@@ -202,11 +202,11 @@ def _attn(self, q, k, v, attention_mask=None, head_mask=None, output_attentions=
def merge_heads(self, x):
x = x.permute(0, 2, 1, 3).contiguous()
new_x_shape = x.size()[:-2] + (x.size(-2) * x.size(-1),)
- return x.view(*new_x_shape) # in Tensorflow implem: fct merge_states
+ return x.view(*new_x_shape) # in Tensorflow implementation: fct merge_states
def split_heads(self, x, k=False):
new_x_shape = x.size()[:-1] + (self.n_head, x.size(-1) // self.n_head)
- x = x.view(*new_x_shape) # in Tensorflow implem: fct split_states
+ x = x.view(*new_x_shape) # in Tensorflow implementation: fct split_states
if k:
return x.permute(0, 2, 3, 1)
else:
@@ -467,7 +467,7 @@ def forward(
raise ValueError("You have to specify either input_ids or inputs_embeds")
if position_ids is None:
- # Code is different from when we had a single embedding matrice from position and token embeddings
+ # Code is different from when we had a single embedding matrix from position and token embeddings
position_ids = self.position_ids[None, : input_shape[-1]]
# Attention mask.
@@ -814,7 +814,7 @@ def forward(
sequence_lengths = -1
logger.warning(
f"{self.__class__.__name__} will not detect padding tokens in `inputs_embeds`. Results may be "
- f"unexpected if using padding tokens in conjuction with `inputs_embeds.`"
+ f"unexpected if using padding tokens in conjunction with `inputs_embeds.`"
)
pooled_logits = logits[range(batch_size), sequence_lengths]
diff --git a/src/transformers/models/openai/modeling_tf_openai.py b/src/transformers/models/openai/modeling_tf_openai.py
index 36679c9643c9..97496ec63a79 100644
--- a/src/transformers/models/openai/modeling_tf_openai.py
+++ b/src/transformers/models/openai/modeling_tf_openai.py
@@ -62,7 +62,7 @@ def __init__(self, nx, n_ctx, config, scale=False, **kwargs):
super().__init__(**kwargs)
n_state = nx # in Attention: n_state=768 (nx=n_embd)
- # [switch nx => n_state from Block to Attention to keep identical to TF implem]
+ # [switch nx => n_state from Block to Attention to keep identical to TF implementation]
assert (
n_state % config.n_head == 0
), f"Hidden dimension {n_state} not dividable by number of heads {config.n_head}"
diff --git a/src/transformers/models/openai/tokenization_openai.py b/src/transformers/models/openai/tokenization_openai.py
index 92d4286c6046..e5bc6b245fb3 100644
--- a/src/transformers/models/openai/tokenization_openai.py
+++ b/src/transformers/models/openai/tokenization_openai.py
@@ -176,7 +176,7 @@ def bpe(self, token):
return word
def _tokenize(self, text):
- """ Tokenize a string. """
+ """Tokenize a string."""
split_tokens = []
if self.fix_text is None:
# Using BERT's BasicTokenizer
@@ -191,7 +191,7 @@ def _tokenize(self, text):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.encoder.get(token, self.encoder.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -199,7 +199,7 @@ def _convert_id_to_token(self, index):
return self.decoder.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = "".join(tokens).replace("", " ").strip()
return out_string
diff --git a/src/transformers/models/pegasus/modeling_pegasus.py b/src/transformers/models/pegasus/modeling_pegasus.py
index 096150e08ee7..66a15964e6a6 100755
--- a/src/transformers/models/pegasus/modeling_pegasus.py
+++ b/src/transformers/models/pegasus/modeling_pegasus.py
@@ -252,9 +252,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -558,7 +558,7 @@ def dummy_inputs(self):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
Pegasus uses the :obj:`pad_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
diff --git a/src/transformers/models/pegasus/modeling_tf_pegasus.py b/src/transformers/models/pegasus/modeling_tf_pegasus.py
index 671a75032ab1..3fadffad18b3 100644
--- a/src/transformers/models/pegasus/modeling_tf_pegasus.py
+++ b/src/transformers/models/pegasus/modeling_tf_pegasus.py
@@ -31,7 +31,7 @@
)
from ...modeling_tf_outputs import (
TFBaseModelOutput,
- TFBaseModelOutputWithPast,
+ TFBaseModelOutputWithPastAndCrossAttentions,
TFSeq2SeqLMOutput,
TFSeq2SeqModelOutput,
)
@@ -409,7 +409,7 @@ def call(
encoder_hidden_states: Optional[tf.Tensor] = None,
encoder_attention_mask: Optional[tf.Tensor] = None,
layer_head_mask: Optional[tf.Tensor] = None,
- encoder_layer_head_mask: Optional[tf.Tensor] = None,
+ cross_attn_layer_head_mask: Optional[tf.Tensor] = None,
past_key_value: Optional[Tuple[tf.Tensor]] = None,
training=False,
) -> Tuple[tf.Tensor, tf.Tensor, Tuple[Tuple[tf.Tensor]]]:
@@ -423,8 +423,8 @@ def call(
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
layer_head_mask (:obj:`tf.Tensor`): mask for attention heads in a given layer of size
`(decoder_attention_heads,)`
- encoder_layer_head_mask (:obj:`tf.Tensor`): mask for encoder attention heads in a given layer of size
- `(encoder_attention_heads,)`
+ cross_attn_layer_head_mask (:obj:`tf.Tensor`): mask for heads of the cross-attention module.
+ `(decoder_attention_heads,)`
past_key_value (:obj:`Tuple(tf.Tensor)`): cached past key and value projection states
"""
residual = hidden_states
@@ -445,17 +445,18 @@ def call(
# Cross-Attention Block
cross_attn_present_key_value = None
+ cross_attn_weights = None
if encoder_hidden_states is not None:
residual = hidden_states
hidden_states = self.encoder_attn_layer_norm(hidden_states)
# cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
- hidden_states, _, cross_attn_present_key_value = self.encoder_attn(
+ hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
hidden_states=hidden_states,
key_value_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
- layer_head_mask=encoder_layer_head_mask,
+ layer_head_mask=cross_attn_layer_head_mask,
past_key_value=cross_attn_past_key_value,
)
hidden_states = self.dropout(hidden_states, training=training)
@@ -476,6 +477,7 @@ def call(
return (
hidden_states,
self_attn_weights,
+ cross_attn_weights,
present_key_value,
)
@@ -592,7 +594,7 @@ def serving(self, inputs):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
Pegasus uses the :obj:`pad_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -603,7 +605,7 @@ def serving(self, inputs):
Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
decoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
@@ -611,6 +613,12 @@ def serving(self, inputs):
- 1 indicates the head is **not masked**,
- 0 indicates the head is **masked**.
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
encoder_outputs (:obj:`tf.FloatTensor`, `optional`):
hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
of shape :obj:`(batch_size, sequence_length, hidden_size)` is a sequence of
@@ -711,7 +719,7 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
@@ -855,7 +863,7 @@ def call(
encoder_hidden_states=None,
encoder_attention_mask=None,
head_mask=None,
- encoder_head_mask=None,
+ cross_attn_head_mask=None,
past_key_values=None,
use_cache=None,
output_attentions=None,
@@ -897,14 +905,13 @@ def call(
Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
- encoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional`):
- Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention
- on hidden heads. Mask values selected in ``[0, 1]``:
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
- 1 indicates the head is **not masked**,
- - 0 indicates the heas is **masked**.
+ - 0 indicates the head is **masked**.
past_key_values (:obj:`Tuple[Tuple[tf.Tensor]]` of length :obj:`config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up
@@ -941,7 +948,7 @@ def call(
encoder_hidden_states=encoder_hidden_states,
encoder_attention_mask=encoder_attention_mask,
head_mask=head_mask,
- encoder_head_mask=encoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
inputs_embeds=inputs_embeds,
past_key_values=past_key_values,
use_cache=use_cache,
@@ -993,19 +1000,21 @@ def call(
hidden_states = self.dropout(hidden_states + positions, training=inputs["training"])
# decoder layers
- all_hidden_states = ()
- all_self_attns = ()
- present_key_values = ()
+ all_hidden_states = () if inputs["output_hidden_states"] else None
+ all_self_attns = () if inputs["output_attentions"] else None
+ all_cross_attns = () if (inputs["output_attentions"] and inputs["encoder_hidden_states"] is not None) else None
+ present_key_values = () if inputs["use_cache"] else None
- # check if head_mask has a correct number of layers specified if desired
+ # check if head_mask and cross_attn_head_mask have a correct number of layers specified if desired
# The tf.debugging asserts are not compliant with XLA then they
# have to be disabled in other modes than eager.
- if inputs["head_mask"] is not None and tf.executing_eagerly():
- tf.debugging.assert_equal(
- shape_list(inputs["head_mask"])[0],
- len(self.layers),
- message=f"The head_mask should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs['head_mask'])[0]}.",
- )
+ for attn_mask in ["head_mask", "cross_attn_head_mask"]:
+ if inputs[attn_mask] is not None and tf.executing_eagerly():
+ tf.debugging.assert_equal(
+ shape_list(inputs[attn_mask])[0],
+ len(self.layers),
+ message=f"The {attn_mask} should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs[attn_mask])[0]}.",
+ )
for idx, decoder_layer in enumerate(self.layers):
# add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
@@ -1018,14 +1027,14 @@ def call(
past_key_value = inputs["past_key_values"][idx] if inputs["past_key_values"] is not None else None
- hidden_states, layer_self_attn, present_key_value = decoder_layer(
+ hidden_states, layer_self_attn, layer_cross_attn, present_key_value = decoder_layer(
hidden_states,
attention_mask=combined_attention_mask,
encoder_hidden_states=inputs["encoder_hidden_states"],
encoder_attention_mask=inputs["encoder_attention_mask"],
layer_head_mask=inputs["head_mask"][idx] if inputs["head_mask"] is not None else None,
- encoder_layer_head_mask=inputs["encoder_head_mask"][idx]
- if inputs["encoder_head_mask"] is not None
+ cross_attn_layer_head_mask=inputs["cross_attn_head_mask"][idx]
+ if inputs["cross_attn_head_mask"] is not None
else None,
past_key_value=past_key_value,
)
@@ -1036,25 +1045,32 @@ def call(
if inputs["output_attentions"]:
all_self_attns += (layer_self_attn,)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns += (layer_cross_attn,)
+
hidden_states = self.layer_norm(hidden_states)
if inputs["output_hidden_states"]:
all_hidden_states += (hidden_states,)
- else:
- all_hidden_states = None
- all_self_attns = list(all_self_attns) if inputs["output_attentions"] else None
+ if inputs["output_attentions"]:
+ all_self_attns = list(all_self_attns)
+
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns = list(all_cross_attns)
- present_key_values = (encoder_hidden_states, present_key_values) if inputs["use_cache"] else None
+ if inputs["use_cache"]:
+ present_key_values = (inputs["encoder_hidden_states"], present_key_values)
if not inputs["return_dict"]:
- return hidden_states, present_key_values, all_hidden_states, all_self_attns
+ return hidden_states, present_key_values, all_hidden_states, all_self_attns, all_cross_attns
else:
- return TFBaseModelOutputWithPast(
+ return TFBaseModelOutputWithPastAndCrossAttentions(
last_hidden_state=hidden_states,
past_key_values=present_key_values,
hidden_states=all_hidden_states,
attentions=all_self_attns,
+ cross_attentions=all_cross_attns,
)
@@ -1101,6 +1117,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1121,6 +1138,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1170,7 +1188,7 @@ def call(
encoder_hidden_states=inputs["encoder_outputs"][0],
encoder_attention_mask=inputs["attention_mask"],
head_mask=inputs["decoder_head_mask"],
- encoder_head_mask=inputs["head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["decoder_inputs_embeds"],
use_cache=inputs["use_cache"],
@@ -1188,6 +1206,7 @@ def call(
past_key_values=decoder_outputs.past_key_values,
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
+ cross_attentions=decoder_outputs.cross_attentions,
encoder_last_hidden_state=inputs["encoder_outputs"].last_hidden_state,
encoder_hidden_states=inputs["encoder_outputs"].hidden_states,
encoder_attentions=inputs["encoder_outputs"].attentions,
@@ -1225,6 +1244,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1245,6 +1265,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1264,6 +1285,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
encoder_outputs=inputs["encoder_outputs"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
@@ -1282,6 +1304,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1290,6 +1313,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
@@ -1344,6 +1368,7 @@ def call(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[TFBaseModelOutput] = None,
past_key_values=None,
inputs_embeds=None,
@@ -1374,6 +1399,7 @@ def call(
decoder_attention_mask=decoder_attention_mask,
head_mask=head_mask,
decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -1407,6 +1433,7 @@ def call(
decoder_attention_mask=inputs["decoder_attention_mask"],
head_mask=inputs["head_mask"],
decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
decoder_inputs_embeds=inputs["decoder_inputs_embeds"],
@@ -1429,6 +1456,7 @@ def call(
past_key_values=outputs.past_key_values, # index 1 of d outputs
decoder_hidden_states=outputs.decoder_hidden_states, # index 2 of d outputs
decoder_attentions=outputs.decoder_attentions, # index 3 of d outputs
+ cross_attentions=outputs.cross_attentions, # index 4 of d outputs
encoder_last_hidden_state=outputs.encoder_last_hidden_state, # index 0 of encoder outputs
encoder_hidden_states=outputs.encoder_hidden_states, # 1 of e out
encoder_attentions=outputs.encoder_attentions, # 2 of e out
@@ -1439,6 +1467,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -1447,6 +1476,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
diff --git a/src/transformers/models/pegasus/tokenization_pegasus.py b/src/transformers/models/pegasus/tokenization_pegasus.py
index 472ca424bbbe..7ced56725489 100644
--- a/src/transformers/models/pegasus/tokenization_pegasus.py
+++ b/src/transformers/models/pegasus/tokenization_pegasus.py
@@ -175,7 +175,7 @@ def _tokenize(self, text, sample=False):
return pieces
def _convert_token_to_id(self, token: str) -> int:
- """ Converts a token (str) to an id using the vocab. """
+ """Converts a token (str) to an id using the vocab."""
if token in self.decoder:
return self.decoder[token]
elif token in self.added_tokens_decoder:
@@ -194,7 +194,7 @@ def _convert_id_to_token(self, index: int) -> str:
return token
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = self.sp_model.decode_pieces(tokens)
return out_string
diff --git a/src/transformers/models/phobert/tokenization_phobert.py b/src/transformers/models/phobert/tokenization_phobert.py
index 3caca9012d23..a07e5bba3a58 100644
--- a/src/transformers/models/phobert/tokenization_phobert.py
+++ b/src/transformers/models/phobert/tokenization_phobert.py
@@ -295,7 +295,7 @@ def _tokenize(self, text):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.encoder.get(token, self.encoder.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -303,7 +303,7 @@ def _convert_id_to_token(self, index):
return self.decoder.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = " ".join(tokens).replace("@@ ", "").strip()
return out_string
diff --git a/src/transformers/models/prophetnet/modeling_prophetnet.py b/src/transformers/models/prophetnet/modeling_prophetnet.py
index ab353b334360..64d8d36e3fd5 100644
--- a/src/transformers/models/prophetnet/modeling_prophetnet.py
+++ b/src/transformers/models/prophetnet/modeling_prophetnet.py
@@ -95,7 +95,7 @@
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
ProphetNet uses the :obj:`eos_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -723,9 +723,9 @@ def forward(
attn_weights = attn_weights + attention_mask
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(batch_size, self.num_attn_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(batch_size * self.num_attn_heads, tgt_len, src_len)
@@ -1243,7 +1243,7 @@ class ProphetNetEncoder(ProphetNetPreTrainedModel):
r"""
word_embeddings (:obj:`torch.nn.Embeddings` of shape :obj:`(config.vocab_size, config.hidden_size)`, `optional`):
The word embedding parameters. This can be used to initialize :class:`~transformers.ProphetNetEncoder` with
- pre-defined word embeddings instead of randomely initialized word embeddings.
+ pre-defined word embeddings instead of randomly initialized word embeddings.
"""
def __init__(self, config: ProphetNetConfig, word_embeddings: nn.Embedding = None):
@@ -1380,7 +1380,7 @@ class ProphetNetDecoder(ProphetNetPreTrainedModel):
r"""
word_embeddings (:obj:`torch.nn.Embeddings` of shape :obj:`(config.vocab_size, config.hidden_size)`, `optional`):
The word embedding parameters. This can be used to initialize :class:`~transformers.ProphetNetEncoder` with
- pre-defined word embeddings instead of randomely initialized word embeddings.
+ pre-defined word embeddings instead of randomly initialized word embeddings.
"""
def __init__(self, config: ProphetNetConfig, word_embeddings: nn.Embedding = None):
diff --git a/src/transformers/models/prophetnet/tokenization_prophetnet.py b/src/transformers/models/prophetnet/tokenization_prophetnet.py
index 25df78162e62..56f26df0e4e0 100644
--- a/src/transformers/models/prophetnet/tokenization_prophetnet.py
+++ b/src/transformers/models/prophetnet/tokenization_prophetnet.py
@@ -172,7 +172,7 @@ def _tokenize(self, text):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.vocab.get(token, self.vocab.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -180,7 +180,7 @@ def _convert_id_to_token(self, index):
return self.ids_to_tokens.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = " ".join(tokens).replace(" ##", "").strip()
return out_string
diff --git a/src/transformers/models/rag/modeling_rag.py b/src/transformers/models/rag/modeling_rag.py
index 7975361749f7..42c2e16d6ca7 100644
--- a/src/transformers/models/rag/modeling_rag.py
+++ b/src/transformers/models/rag/modeling_rag.py
@@ -1543,7 +1543,6 @@ def extend_enc_output(tensor, num_beams=None):
raise ValueError("`num_return_sequences` has to be smaller or equal to `num_beams`.")
beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=max_length,
num_beams=num_beams,
device=self.device,
length_penalty=length_penalty,
diff --git a/src/transformers/models/rag/modeling_tf_rag.py b/src/transformers/models/rag/modeling_tf_rag.py
index 4e80f8fd08ba..4d452b635998 100644
--- a/src/transformers/models/rag/modeling_tf_rag.py
+++ b/src/transformers/models/rag/modeling_tf_rag.py
@@ -285,7 +285,7 @@ def from_pretrained_question_encoder_generator(
>>> # load retriever
>>> retriever = RagRetriever.from_pretrained(PATH, index_name="exact", use_dummy_dataset=True)
- >>> # load fine-tuned model with retriver
+ >>> # load fine-tuned model with retriever
>>> model = TFRagModel.from_pretrained("./rag", retriever=retriever)
"""
diff --git a/src/transformers/models/rag/retrieval_rag.py b/src/transformers/models/rag/retrieval_rag.py
index 8e2676298716..c0e6f30072d6 100644
--- a/src/transformers/models/rag/retrieval_rag.py
+++ b/src/transformers/models/rag/retrieval_rag.py
@@ -234,7 +234,7 @@ class CanonicalHFIndex(HFIndexBase):
Args:
vector_size (:obj:`int`): the dimension of the passages embeddings used by the index
dataset_name (:obj:`str`, optional, defaults to ``wiki_dpr``):
- A datatset identifier of the indexed dataset on HuggingFace AWS bucket (list all available datasets and ids
+ A dataset identifier of the indexed dataset on HuggingFace AWS bucket (list all available datasets and ids
with ``datasets.list_datasets()``).
dataset_split (:obj:`str`, optional, defaults to ``train``)
Which split of the ``dataset`` to load.
@@ -442,7 +442,7 @@ def save_pretrained(self, save_directory):
def init_retrieval(self):
"""
- Retriever initalization function. It loads the index into memory.
+ Retriever initialization function. It loads the index into memory.
"""
logger.info("initializing retrieval")
diff --git a/src/transformers/models/reformer/modeling_reformer.py b/src/transformers/models/reformer/modeling_reformer.py
index 28f0fdd08ed5..0420a1c8ee4f 100755
--- a/src/transformers/models/reformer/modeling_reformer.py
+++ b/src/transformers/models/reformer/modeling_reformer.py
@@ -612,7 +612,7 @@ def _hash_vectors(self, vectors, num_hashes, attention_mask, increase_num_bucket
if isinstance(self.num_buckets, int):
assert (
self.num_buckets % 2 == 0
- ), f"There should be an even number of bucktes, but `self.num_bucktes`: {self.num_buckets}"
+ ), f"There should be an even number of buckets, but `self.num_buckets`: {self.num_buckets}"
rotation_size = self.num_buckets
num_buckets = self.num_buckets
else:
@@ -1779,7 +1779,7 @@ def dummy_inputs(self):
return dummy_inputs
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, AxialPositionEmbeddings):
for weight in module.weights:
torch.nn.init.normal_(weight, std=self.config.axial_norm_std)
diff --git a/src/transformers/models/reformer/tokenization_reformer.py b/src/transformers/models/reformer/tokenization_reformer.py
index c933d0cbc763..535a93a31ac0 100644
--- a/src/transformers/models/reformer/tokenization_reformer.py
+++ b/src/transformers/models/reformer/tokenization_reformer.py
@@ -115,7 +115,7 @@ def _tokenize(self, text, sample=False):
return pieces
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.sp_model.piece_to_id(token)
def _convert_id_to_token(self, index):
@@ -125,7 +125,7 @@ def _convert_id_to_token(self, index):
return token
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = self.sp_model.decode_pieces(tokens)
return out_string
diff --git a/src/transformers/models/retribert/modeling_retribert.py b/src/transformers/models/retribert/modeling_retribert.py
index 0b6023e7bc7e..2507688209e7 100644
--- a/src/transformers/models/retribert/modeling_retribert.py
+++ b/src/transformers/models/retribert/modeling_retribert.py
@@ -50,7 +50,7 @@ class RetriBertPreTrainedModel(PreTrainedModel):
base_model_prefix = "retribert"
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
if module.bias is not None:
diff --git a/src/transformers/models/roberta/modeling_roberta.py b/src/transformers/models/roberta/modeling_roberta.py
index f7a73b336c79..274833d0507d 100644
--- a/src/transformers/models/roberta/modeling_roberta.py
+++ b/src/transformers/models/roberta/modeling_roberta.py
@@ -574,7 +574,7 @@ class RobertaPreTrainedModel(PreTrainedModel):
# Copied from transformers.models.bert.modeling_bert.BertPreTrainedModel._init_weights
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/roberta/tokenization_roberta_fast.py b/src/transformers/models/roberta/tokenization_roberta_fast.py
index c450be4a29f0..243cac19d1c4 100644
--- a/src/transformers/models/roberta/tokenization_roberta_fast.py
+++ b/src/transformers/models/roberta/tokenization_roberta_fast.py
@@ -179,7 +179,7 @@ def mask_token(self) -> str:
:obj:`str`: Mask token, to use when training a model with masked-language modeling. Log an error if used while
not having been set.
- Roberta tokenizer has a special mask token to be usble in the fill-mask pipeline. The mask token will greedily
+ Roberta tokenizer has a special mask token to be usable in the fill-mask pipeline. The mask token will greedily
comprise the space before the ``.
"""
if self._mask_token is None and self.verbose:
diff --git a/src/transformers/models/speech_to_text/modeling_speech_to_text.py b/src/transformers/models/speech_to_text/modeling_speech_to_text.py
index 4b61f5ae7dd3..ff50202b356c 100755
--- a/src/transformers/models/speech_to_text/modeling_speech_to_text.py
+++ b/src/transformers/models/speech_to_text/modeling_speech_to_text.py
@@ -319,9 +319,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -559,7 +559,7 @@ def _get_subsampled_output_lengths(self, input_lengths: torch.LongTensor):
return input_lengths
def _get_subsampled_encoder_attn_mask(self, attention_mask):
- # generate creates 3D attention mask, becuase of the shape of input_features
+ # generate creates 3D attention mask, because of the shape of input_features
# convert it to 2D if thats the case
if len(attention_mask.shape) > 2:
attention_mask = attention_mask[:, :, -1]
@@ -616,7 +616,7 @@ def _get_subsampled_encoder_attn_mask(self, attention_mask):
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
SpeechToText uses the :obj:`eos_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
diff --git a/src/transformers/models/squeezebert/modeling_squeezebert.py b/src/transformers/models/squeezebert/modeling_squeezebert.py
index 09dcd680bbb4..ce7d18808dc3 100644
--- a/src/transformers/models/squeezebert/modeling_squeezebert.py
+++ b/src/transformers/models/squeezebert/modeling_squeezebert.py
@@ -431,7 +431,7 @@ class SqueezeBertPreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, (nn.Linear, nn.Conv1d)):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/t5/modeling_t5.py b/src/transformers/models/t5/modeling_t5.py
index 013f291c5ba0..746f4c389482 100644
--- a/src/transformers/models/t5/modeling_t5.py
+++ b/src/transformers/models/t5/modeling_t5.py
@@ -389,7 +389,7 @@ def _relative_position_bucket(relative_position, bidirectional=True, num_buckets
return relative_buckets
def compute_bias(self, query_length, key_length):
- """ Compute binned relative position bias """
+ """Compute binned relative position bias"""
context_position = torch.arange(query_length, dtype=torch.long)[:, None]
memory_position = torch.arange(key_length, dtype=torch.long)[None, :]
relative_position = memory_position - context_position # shape (query_length, key_length)
@@ -436,15 +436,15 @@ def forward(
key_length = real_seq_length if key_value_states is None else key_value_states.shape[1]
def shape(states):
- """ projection """
+ """projection"""
return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2)
def unshape(states):
- """ reshape """
+ """reshape"""
return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim)
def project(hidden_states, proj_layer, key_value_states, past_key_value):
- """ projects hidden states correctly to key/query states """
+ """projects hidden states correctly to key/query states"""
if key_value_states is None:
# self-attn
# (batch_size, n_heads, seq_length, dim_per_head)
@@ -718,7 +718,7 @@ def dummy_inputs(self):
return dummy_inputs
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
factor = self.config.initializer_factor # Used for testing weights initialization
if isinstance(module, T5LayerNorm):
module.weight.data.fill_(factor * 1.0)
@@ -1063,7 +1063,7 @@ def forward(
:meth:`transformers.PreTrainedTokenizer.encode` and :meth:`transformers.PreTrainedTokenizer.__call__` for
details.
- `What are input IDs? <../glossary.html#input-ids>`__
+ `What are decoder input IDs? <../glossary.html#decoder-input-ids>`__
T5 uses the :obj:`pad_token_id` as the starting token for :obj:`decoder_input_ids` generation. If
:obj:`past_key_values` is used, optionally only the last :obj:`decoder_input_ids` have to be input (see
@@ -1172,7 +1172,7 @@ def forward(
Whether or not to return a :class:`~transformers.file_utils.ModelOutput` instead of a plain tuple.
"""
-# Warning messafe for FutureWarning: head_mask was separated into two input args - head_mask, decoder_head_mask
+# Warning message for FutureWarning: head_mask was separated into two input args - head_mask, decoder_head_mask
__HEAD_MASK_WARNING_MSG = """
The input argument `head_mask` was split into two arguments `head_mask` and `decoder_head_mask`. Currently,
`decoder_head_mask` is set to copy `head_mask`, but this feature is deprecated and will be removed in future versions.
diff --git a/src/transformers/models/t5/modeling_tf_t5.py b/src/transformers/models/t5/modeling_tf_t5.py
index d1a9f0aaca1f..4d70cb2c3e5f 100644
--- a/src/transformers/models/t5/modeling_tf_t5.py
+++ b/src/transformers/models/t5/modeling_tf_t5.py
@@ -80,7 +80,7 @@ def __init__(self, epsilon=1e-6, **kwargs):
self.variance_epsilon = epsilon
def build(self, input_shape):
- """Build shared word embedding layer """
+ """Build shared word embedding layer"""
self.weight = self.add_weight("weight", shape=(input_shape[-1],), initializer="ones")
super().build(input_shape)
@@ -230,7 +230,7 @@ def _relative_position_bucket(relative_position, bidirectional=True, num_buckets
return relative_buckets
def compute_bias(self, query_length, key_length):
- """ Compute binned relative position bias """
+ """Compute binned relative position bias"""
context_position = tf.range(query_length)[:, None]
memory_position = tf.range(key_length)[None, :]
relative_position = memory_position - context_position # shape (query_length, key_length)
@@ -279,17 +279,17 @@ def call(
key_length = real_seq_length if key_value_states is None else shape_list(key_value_states)[1]
def shape(hidden_states):
- """ projection """
+ """projection"""
return tf.transpose(
tf.reshape(hidden_states, (batch_size, -1, self.n_heads, self.key_value_proj_dim)), perm=(0, 2, 1, 3)
)
def unshape(hidden_states):
- """ compute context """
+ """compute context"""
return tf.reshape(tf.transpose(hidden_states, perm=(0, 2, 1, 3)), (batch_size, -1, self.inner_dim))
def project(hidden_states, proj_layer, key_value_states, past_key_value):
- """ projects hidden states correctly to key/query states """
+ """projects hidden states correctly to key/query states"""
if key_value_states is None:
# self-attn
# (batch_size, n_heads, seq_length, dim_per_head)
@@ -637,7 +637,7 @@ def call(
raise ValueError(f"You have to specify either {err_msg_prefix}inputs or {err_msg_prefix}inputs_embeds")
if inputs["inputs_embeds"] is None:
- assert self.embed_tokens is not None, "You have to intialize the model with valid token embeddings"
+ assert self.embed_tokens is not None, "You have to initialize the model with valid token embeddings"
inputs["inputs_embeds"] = self.embed_tokens(inputs["input_ids"])
batch_size, seq_length = input_shape
diff --git a/src/transformers/models/t5/tokenization_t5.py b/src/transformers/models/t5/tokenization_t5.py
index 90a0159aefb7..a069cf448814 100644
--- a/src/transformers/models/t5/tokenization_t5.py
+++ b/src/transformers/models/t5/tokenization_t5.py
@@ -243,7 +243,7 @@ def _tokenize(self, text, sample=False):
return pieces
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
if token.startswith("", token)
num = int(match.group(1))
@@ -259,7 +259,7 @@ def _convert_id_to_token(self, index):
return token
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
current_sub_tokens = []
out_string = ""
for token in tokens:
diff --git a/src/transformers/models/tapas/modeling_tapas.py b/src/transformers/models/tapas/modeling_tapas.py
index 5bfca58596c4..fb49cb9b2db1 100644
--- a/src/transformers/models/tapas/modeling_tapas.py
+++ b/src/transformers/models/tapas/modeling_tapas.py
@@ -699,7 +699,7 @@ class TapasPreTrainedModel(PreTrainedModel):
# Copied from transformers.models.bert.modeling_bert.BertPreTrainedModel._init_weights
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
@@ -2021,7 +2021,7 @@ def _calculate_aggregate_mask(answer, pooled_output, cell_selection_preference,
apply to numbers. If the answer is a number but does not appear in the table then we must use some aggregation
case. The ambiguous case is when the answer is a number that also appears in the table. In this case we use the
aggregation function probabilities predicted by the model to decide whether to select or aggregate. The threshold
- for this is a hyperparameter `cell_selection_preference
+ for this is a hyperparameter `cell_selection_preference`
Args:
answer (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, )`):
@@ -2041,7 +2041,7 @@ def _calculate_aggregate_mask(answer, pooled_output, cell_selection_preference,
aggregate_mask_init = torch.logical_not(torch.isnan(answer)).type(torch.FloatTensor).to(answer.device)
logits_aggregation = aggregation_classifier(pooled_output)
dist_aggregation = torch.distributions.categorical.Categorical(logits=logits_aggregation)
- # Index 0 correponds to "no aggregation".
+ # Index 0 corresponds to "no aggregation".
aggregation_ops_total_mass = torch.sum(dist_aggregation.probs[:, 1:], dim=1)
# Cell selection examples according to current model.
@@ -2126,7 +2126,7 @@ def _calculate_aggregation_loss_unknown(logits_aggregation, aggregate_mask):
answer supervision) per example.
"""
dist_aggregation = torch.distributions.categorical.Categorical(logits=logits_aggregation)
- # Index 0 correponds to "no aggregation".
+ # Index 0 corresponds to "no aggregation".
aggregation_ops_total_mass = torch.sum(dist_aggregation.probs[:, 1:], dim=1)
# Predict some aggregation in case of an answer that needs aggregation.
# This increases the probability of all aggregation functions, in a way
diff --git a/src/transformers/models/tapas/tokenization_tapas.py b/src/transformers/models/tapas/tokenization_tapas.py
index 3d1e82ac5180..6e9f439ea124 100644
--- a/src/transformers/models/tapas/tokenization_tapas.py
+++ b/src/transformers/models/tapas/tokenization_tapas.py
@@ -172,8 +172,9 @@ def whitespace_tokenize(text):
length is required by one of the truncation/padding parameters. If the model has no specific maximum
input length (like XLNet) truncation/padding to a maximum length will be deactivated.
is_split_into_words (:obj:`bool`, `optional`, defaults to :obj:`False`):
- Whether or not the input is already pre-tokenized (e.g., split into words), in which case the tokenizer
- will skip the pre-tokenization step. This is useful for NER or token classification.
+ Whether or not the input is already pre-tokenized (e.g., split into words). If set to :obj:`True`, the
+ tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
+ which it will tokenize. This is useful for NER or token classification.
pad_to_multiple_of (:obj:`int`, `optional`):
If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
@@ -374,7 +375,7 @@ def _tokenize(self, text):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.vocab.get(token, self.vocab.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -382,7 +383,7 @@ def _convert_id_to_token(self, index):
return self.ids_to_tokens.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = " ".join(tokens).replace(" ##", "").strip()
return out_string
@@ -2357,7 +2358,7 @@ def _get_numeric_value_from_date(date, mask):
def _get_span_length_key(span):
- """Sorts span by decreasing length first and incresing first index second."""
+ """Sorts span by decreasing length first and increasing first index second."""
return span[1] - span[0], -span[0]
diff --git a/src/transformers/models/transfo_xl/tokenization_transfo_xl.py b/src/transformers/models/transfo_xl/tokenization_transfo_xl.py
index 9b185ecdd1e1..e380197a5f1b 100644
--- a/src/transformers/models/transfo_xl/tokenization_transfo_xl.py
+++ b/src/transformers/models/transfo_xl/tokenization_transfo_xl.py
@@ -292,7 +292,7 @@ def _build_from_file(self, vocab_file):
elif "" in self.sym2idx:
self.unk_idx = self.sym2idx[""]
else:
- raise ValueError("No token in vocabulary")
+ raise ValueError("No token in vocabulary")
def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
if os.path.isdir(save_directory):
@@ -434,7 +434,7 @@ def _convert_id_to_token(self, idx):
return self.idx2sym[idx]
def _convert_token_to_id(self, sym):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
if sym in self.sym2idx:
return self.sym2idx[sym]
else:
diff --git a/src/transformers/models/vit/modeling_vit.py b/src/transformers/models/vit/modeling_vit.py
index 559dfff83c3c..3584813db62a 100644
--- a/src/transformers/models/vit/modeling_vit.py
+++ b/src/transformers/models/vit/modeling_vit.py
@@ -372,7 +372,7 @@ class ViTPreTrainedModel(PreTrainedModel):
base_model_prefix = "vit"
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, (nn.Linear, nn.Conv2d)):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/wav2vec2/modeling_wav2vec2.py b/src/transformers/models/wav2vec2/modeling_wav2vec2.py
index ba548dc3d8ea..98123bdd310e 100755
--- a/src/transformers/models/wav2vec2/modeling_wav2vec2.py
+++ b/src/transformers/models/wav2vec2/modeling_wav2vec2.py
@@ -382,9 +382,9 @@ def forward(
attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
if output_attentions:
- # this operation is a bit akward, but it's required to
+ # this operation is a bit awkward, but it's required to
# make sure that attn_weights keeps its gradient.
- # In order to do so, attn_weights have to reshaped
+ # In order to do so, attn_weights have to be reshaped
# twice and have to be reused in the following
attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)
@@ -680,7 +680,7 @@ class Wav2Vec2PreTrainedModel(PreTrainedModel):
_keys_to_ignore_on_load_missing = [r"position_ids"]
def _init_weights(self, module):
- """ Initialize the weights """
+ """Initialize the weights"""
if isinstance(module, nn.Linear):
# Slightly different from the TF version which uses truncated_normal for initialization
# cf https://github.com/pytorch/pytorch/pull/5617
diff --git a/src/transformers/models/wav2vec2/processing_wav2vec2.py b/src/transformers/models/wav2vec2/processing_wav2vec2.py
index 88e3235abd7d..bafbcdebbc75 100644
--- a/src/transformers/models/wav2vec2/processing_wav2vec2.py
+++ b/src/transformers/models/wav2vec2/processing_wav2vec2.py
@@ -111,7 +111,7 @@ def __call__(self, *args, **kwargs):
When used in normal mode, this method forwards all its arguments to Wav2Vec2FeatureExtractor's
:meth:`~transformers.Wav2Vec2FeatureExtractor.__call__` and returns its output. If used in the context
:meth:`~transformers.Wav2Vec2Processor.as_target_processor` this method forwards all its arguments to
- Wav2Vec2CTCTokenizer's :meth:`~transformers.Wav2Vec2CTCTokenizer.__call__`. Please refer to the doctsring of
+ Wav2Vec2CTCTokenizer's :meth:`~transformers.Wav2Vec2CTCTokenizer.__call__`. Please refer to the docstring of
the above two methods for more information.
"""
return self.current_processor(*args, **kwargs)
diff --git a/src/transformers/models/xlm/modeling_tf_xlm.py b/src/transformers/models/xlm/modeling_tf_xlm.py
index 6bac6f597c3f..0ae3ac2a2472 100644
--- a/src/transformers/models/xlm/modeling_tf_xlm.py
+++ b/src/transformers/models/xlm/modeling_tf_xlm.py
@@ -151,11 +151,11 @@ def call(self, input, mask, kv, cache, head_mask, output_attentions, training=Fa
mask_reshape = (bs, 1, qlen, klen) if len(shape_list(mask)) == 3 else (bs, 1, 1, klen)
def shape(x):
- """ projection """
+ """projection"""
return tf.transpose(tf.reshape(x, (bs, -1, self.n_heads, dim_per_head)), perm=(0, 2, 1, 3))
def unshape(x):
- """ compute context """
+ """compute context"""
return tf.reshape(tf.transpose(x, perm=(0, 2, 1, 3)), (bs, -1, self.n_heads * dim_per_head))
q = shape(self.q_lin(input)) # (bs, n_heads, qlen, dim_per_head)
diff --git a/src/transformers/models/xlm/modeling_xlm.py b/src/transformers/models/xlm/modeling_xlm.py
index a4a6c0dd08a3..9ce0e5558ec0 100755
--- a/src/transformers/models/xlm/modeling_xlm.py
+++ b/src/transformers/models/xlm/modeling_xlm.py
@@ -159,11 +159,11 @@ def forward(self, input, mask, kv=None, cache=None, head_mask=None, output_atten
mask_reshape = (bs, 1, qlen, klen) if mask.dim() == 3 else (bs, 1, 1, klen)
def shape(x):
- """ projection """
+ """projection"""
return x.view(bs, -1, self.n_heads, dim_per_head).transpose(1, 2)
def unshape(x):
- """ compute context """
+ """compute context"""
return x.transpose(1, 2).contiguous().view(bs, -1, self.n_heads * dim_per_head)
q = shape(self.q_lin(input)) # (bs, n_heads, qlen, dim_per_head)
@@ -251,7 +251,7 @@ def dummy_inputs(self):
return {"input_ids": inputs_list, "attention_mask": attns_list, "langs": langs_list}
def _init_weights(self, module):
- """ Initialize the weights. """
+ """Initialize the weights."""
if isinstance(module, nn.Embedding):
if self.config is not None and self.config.embed_init_std is not None:
nn.init.normal_(module.weight, mean=0, std=self.config.embed_init_std)
@@ -348,7 +348,7 @@ class XLMForQuestionAnsweringOutput(ModelOutput):
A parallel sequence of tokens to be used to indicate the language of each token in the input. Indices are
languages ids which can be obtained from the language names by using two conversion mappings provided in
the configuration of the model (only provided for multilingual models). More precisely, the `language name
- to language id` mapping is in :obj:`model.config.lang2id` (which is a dictionary strring to int) and the
+ to language id` mapping is in :obj:`model.config.lang2id` (which is a dictionary string to int) and the
`language id to language name` mapping is in :obj:`model.config.id2lang` (dictionary int to string).
See usage examples detailed in the :doc:`multilingual documentation <../multilingual>`.
@@ -1188,7 +1188,7 @@ def __init__(self, config, *inputs, **kwargs):
self.init_weights()
- @add_start_docstrings_to_model_forward(XLM_INPUTS_DOCSTRING.format("batch_size, num_choicec, sequence_length"))
+ @add_start_docstrings_to_model_forward(XLM_INPUTS_DOCSTRING.format("batch_size, num_choices, sequence_length"))
@add_code_sample_docstrings(
tokenizer_class=_TOKENIZER_FOR_DOC,
checkpoint=_CHECKPOINT_FOR_DOC,
diff --git a/src/transformers/models/xlm/tokenization_xlm.py b/src/transformers/models/xlm/tokenization_xlm.py
index 95730451fddd..dbf097992eb6 100644
--- a/src/transformers/models/xlm/tokenization_xlm.py
+++ b/src/transformers/models/xlm/tokenization_xlm.py
@@ -749,8 +749,8 @@ def bpe(self, token):
def _tokenize(self, text, lang="en", bypass_tokenizer=False):
"""
- Tokenize a string given language code. For Chinese, Japanese and Thai, we use a language specific
- tokenizerself. Otherwise, we use Moses.
+ Tokenize a string given language code. For Chinese, Japanese and Thai, we use a language specific tokenizer.
+ Otherwise, we use Moses.
Details of tokenization:
@@ -847,7 +847,7 @@ def _tokenize(self, text, lang="en", bypass_tokenizer=False):
return split_tokens
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.encoder.get(token, self.encoder.get(self.unk_token))
def _convert_id_to_token(self, index):
@@ -855,7 +855,7 @@ def _convert_id_to_token(self, index):
return self.decoder.get(index, self.unk_token)
def convert_tokens_to_string(self, tokens):
- """ Converts a sequence of tokens (string) in a single string. """
+ """Converts a sequence of tokens (string) in a single string."""
out_string = "".join(tokens).replace("", " ").strip()
return out_string
diff --git a/src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py b/src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py
index 188292ed1775..9c2d90914a6d 100644
--- a/src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py
+++ b/src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py
@@ -245,7 +245,7 @@ def _tokenize(self, text):
return self.sp_model.EncodeAsPieces(text)
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
if token in self.fairseq_tokens_to_ids:
return self.fairseq_tokens_to_ids[token]
spm_id = self.sp_model.PieceToId(token)
diff --git a/src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py b/src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py
index 877bfaf1d136..cda78e900df7 100644
--- a/src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py
+++ b/src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py
@@ -270,7 +270,7 @@ def _tokenize(self, text):
return self.sp_model.encode(text, out_type=str)
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
if token in self.fairseq_tokens_to_ids:
return self.fairseq_tokens_to_ids[token]
spm_id = self.sp_model.PieceToId(token)
diff --git a/src/transformers/models/xlnet/modeling_tf_xlnet.py b/src/transformers/models/xlnet/modeling_tf_xlnet.py
index 215a5165615a..bc66d326c4b2 100644
--- a/src/transformers/models/xlnet/modeling_tf_xlnet.py
+++ b/src/transformers/models/xlnet/modeling_tf_xlnet.py
@@ -1113,7 +1113,7 @@ class TFXLNetForQuestionAnsweringSimpleOutput(ModelOutput):
Mask values selected in ``[0, 1]``:
- 1 for tokens that are **masked**,
- - 0 for tokens that are **not maked**.
+ - 0 for tokens that are **not masked**.
You can only uses one of :obj:`input_mask` and :obj:`attention_mask`.
head_mask (:obj:`Numpy array` or :obj:`tf.Tensor` of shape :obj:`(num_heads,)` or :obj:`(num_layers, num_heads)`, `optional`):
diff --git a/src/transformers/models/xlnet/tokenization_xlnet.py b/src/transformers/models/xlnet/tokenization_xlnet.py
index 97fd542c2812..5137bcfee3b8 100644
--- a/src/transformers/models/xlnet/tokenization_xlnet.py
+++ b/src/transformers/models/xlnet/tokenization_xlnet.py
@@ -189,7 +189,7 @@ def preprocess_text(self, inputs):
return outputs
def _tokenize(self, text, sample=False):
- """ Tokenize a string. """
+ """Tokenize a string."""
text = self.preprocess_text(text)
if not sample:
@@ -213,7 +213,7 @@ def _tokenize(self, text, sample=False):
return new_pieces
def _convert_token_to_id(self, token):
- """ Converts a token (str) in an id using the vocab. """
+ """Converts a token (str) in an id using the vocab."""
return self.sp_model.PieceToId(token)
def _convert_id_to_token(self, index):
diff --git a/src/transformers/optimization_tf.py b/src/transformers/optimization_tf.py
index 77cd0d1c6064..8e369223df1b 100644
--- a/src/transformers/optimization_tf.py
+++ b/src/transformers/optimization_tf.py
@@ -182,7 +182,7 @@ class AdamWeightDecay(tf.keras.optimizers.Adam):
name (:obj:`str`, `optional`, defaults to 'AdamWeightDecay'):
Optional name for the operations created when applying gradients.
kwargs:
- Keyward arguments. Allowed to be {``clipnorm``, ``clipvalue``, ``lr``, ``decay``}. ``clipnorm`` is clip
+ Keyword arguments. Allowed to be {``clipnorm``, ``clipvalue``, ``lr``, ``decay``}. ``clipnorm`` is clip
gradients by norm; ``clipvalue`` is clip gradients by value, ``decay`` is included for backward
compatibility to allow time inverse decay of learning rate. ``lr`` is included for backward compatibility,
recommended to use ``learning_rate`` instead.
diff --git a/src/transformers/pipelines/conversational.py b/src/transformers/pipelines/conversational.py
index c77d2141c955..ddbb0a260cd4 100644
--- a/src/transformers/pipelines/conversational.py
+++ b/src/transformers/pipelines/conversational.py
@@ -128,7 +128,7 @@ def iter_texts(self):
"""
Iterates over all blobs of the conversation.
- Retuns: Iterator of (is_user, text_chunk) in chronological order of the conversation. ``is_user`` is a
+ Returns: Iterator of (is_user, text_chunk) in chronological order of the conversation. ``is_user`` is a
:obj:`bool`, ``text_chunks`` is a :obj:`str`.
"""
for user_input, generated_response in zip(self.past_user_inputs, self.generated_responses):
diff --git a/src/transformers/pipelines/text2text_generation.py b/src/transformers/pipelines/text2text_generation.py
index 7a6564aaa452..96aaf3d19fb8 100644
--- a/src/transformers/pipelines/text2text_generation.py
+++ b/src/transformers/pipelines/text2text_generation.py
@@ -48,7 +48,7 @@ def __init__(self, *args, **kwargs):
def check_inputs(self, input_length: int, min_length: int, max_length: int):
"""
- Checks wether there might be something wrong with given input with regard to the model.
+ Checks whether there might be something wrong with given input with regard to the model.
"""
return True
@@ -204,7 +204,7 @@ def __call__(self, *args, **kwargs):
def check_inputs(self, input_length: int, min_length: int, max_length: int) -> bool:
"""
- Checks wether there might be something wrong with given input with regard to the model.
+ Checks whether there might be something wrong with given input with regard to the model.
"""
if input_length < min_length // 2:
logger.warning(
diff --git a/src/transformers/pipelines/token_classification.py b/src/transformers/pipelines/token_classification.py
index d9431c0cb78e..f4254dc6a808 100644
--- a/src/transformers/pipelines/token_classification.py
+++ b/src/transformers/pipelines/token_classification.py
@@ -2,7 +2,7 @@
import numpy as np
-from ..file_utils import add_end_docstrings, is_tf_available, is_torch_available
+from ..file_utils import AggregationStrategy, add_end_docstrings, is_tf_available, is_torch_available
from ..modelcard import ModelCard
from ..models.bert.tokenization_bert import BasicTokenizer
from ..tokenization_utils import PreTrainedTokenizer
@@ -85,6 +85,7 @@ def __init__(
ignore_labels=["O"],
task: str = "",
grouped_entities: bool = False,
+ aggregation_strategy: Union[str, AggregationStrategy] = AggregationStrategy.FIRST,
ignore_subwords: bool = False,
):
super().__init__(
@@ -107,6 +108,10 @@ def __init__(
self._args_parser = args_parser
self.ignore_labels = ignore_labels
self.grouped_entities = grouped_entities
+ if isinstance(aggregation_strategy, str):
+ self.aggregation_strategy = AggregationStrategy[aggregation_strategy.upper()]
+ else:
+ self.aggregation_strategy = aggregation_strategy
self.ignore_subwords = ignore_subwords
if self.ignore_subwords and not self.tokenizer.is_fast:
@@ -177,47 +182,46 @@ def __call__(self, inputs: Union[str, List[str]], **kwargs):
input_ids = tokens["input_ids"].cpu().numpy()[0]
score = np.exp(entities) / np.exp(entities).sum(-1, keepdims=True)
- labels_idx = score.argmax(axis=-1)
entities = []
- # Filter to labels not in `self.ignore_labels`
# Filter special_tokens
- filtered_labels_idx = [
- (idx, label_idx)
- for idx, label_idx in enumerate(labels_idx)
- if (self.model.config.id2label[label_idx] not in self.ignore_labels) and not special_tokens_mask[idx]
- ]
+ filtered_labels_idx = [idx for idx in range(score.shape[0]) if not special_tokens_mask[idx]]
- for idx, label_idx in filtered_labels_idx:
+ for idx in filtered_labels_idx:
+ entity = {}
if offset_mapping is not None:
start_ind, end_ind = offset_mapping[idx]
+ entity["start"], entity["end"] = (start_ind, end_ind)
word_ref = sentence[start_ind:end_ind]
word = self.tokenizer.convert_ids_to_tokens([int(input_ids[idx])])[0]
- is_subword = len(word_ref) != len(word)
+ entity["word"] = word
+ entity["is_subword"] = len(word_ref) != len(word)
if int(input_ids[idx]) == self.tokenizer.unk_token_id:
- word = word_ref
- is_subword = False
+ entity["word"] = word_ref
+ entity["is_subword"] = False
else:
- word = self.tokenizer.convert_ids_to_tokens(int(input_ids[idx]))
-
- start_ind = None
- end_ind = None
+ entity["word"] = self.tokenizer.convert_ids_to_tokens(int(input_ids[idx]))
- entity = {
- "word": word,
- "score": score[idx][label_idx].item(),
- "entity": self.model.config.id2label[label_idx],
- "index": idx,
- "start": start_ind,
- "end": end_ind,
- }
+ entity["start"] = None
+ entity["end"] = None
- if self.grouped_entities and self.ignore_subwords:
- entity["is_subword"] = is_subword
+ entity["score"] = score[idx]
+ entity["index"] = idx
entities += [entity]
+ if self.ignore_subwords:
+ entities = self.set_subwords_label(entities)
+ else:
+ for entity in entities:
+ label_idx = entity["score"].argmax()
+ label = self.model.config.id2label[label_idx]
+ entity["entity"] = label
+ entity["score"] = entity["score"][label_idx].item()
+
+ # filter out ignored labels
+ entities = [entity for entity in entities if entity["entity"] not in self.ignore_labels]
if self.grouped_entities:
answers += [self.group_entities(entities)]
# Append ungrouped entities
@@ -228,6 +232,65 @@ def __call__(self, inputs: Union[str, List[str]], **kwargs):
return answers[0]
return answers
+ def set_subwords_label(self, entities: List[dict]) -> List[dict]:
+ strategy = self.aggregation_strategy
+
+ def set_labels(sub_words: List[dict]) -> dict:
+ scores = np.stack([sub["score"] for sub in sub_words])
+ if strategy == AggregationStrategy.FIRST:
+ # get label of first sub-word
+ max_label_idx = scores[0].argmax()
+ score = scores[0][max_label_idx]
+ label = self.model.config.id2label[max_label_idx]
+ elif strategy == AggregationStrategy.MAX:
+ max_label_idx = np.unravel_index(np.argmax(scores, axis=None), scores.shape)[1]
+ score = scores.max()
+ label = self.model.config.id2label[max_label_idx]
+ elif strategy == AggregationStrategy.AVERAGE:
+ avg_scores = np.mean(scores, axis=0)
+ max_label_idx = avg_scores.argmax()
+ score = avg_scores[max_label_idx]
+ label = self.model.config.id2label[max_label_idx]
+ else:
+ raise ValueError(f"Invalid value {strategy} for option `aggregation_strategy`")
+
+ for idx, sub in enumerate(sub_words):
+ sub["entity"] = label
+ sub["score"] = score.item()
+
+ return sub_words
+
+ subword_indices = np.where([entity["is_subword"] for entity in entities])[0]
+ if subword_indices.size == 0:
+ adjacent_subwords = []
+ else:
+ # find non-consecutive indices to identify separate clusters of subwords
+ cluster_edges = np.where(np.diff(subword_indices) != 1)[0]
+ # Sets of adjacent subwords indices, e.g.
+ # ['Sir', 'Test', '##y', 'M', '##c', '##T', '##est', 'is', 'test', '##iful']
+ # --> [[2],[4,5,6],[9]]
+ adjacent_subwords = np.split(subword_indices, cluster_edges + 1) # shift edge by 1
+
+ word_indices = []
+ start = 0
+ for subwords in adjacent_subwords:
+ root_word = subwords[0] - 1
+ word_indices += [[idx] for idx in range(start, root_word)]
+ word_indices += [[root_word] + list(subwords)]
+ start = subwords[-1] + 1
+ word_indices += [[idx] for idx in range(start, len(entities))]
+
+ entities_with_label = []
+ for word_idx in word_indices:
+ subwords = [entities[idx] for idx in word_idx]
+ subwords = set_labels(subwords)
+ if self.ignore_subwords and len(subwords) > 1:
+ subwords[0]["word"] += "".join([sub["word"].split("##")[1] for sub in subwords[1:]])
+ subwords = [subwords[0]]
+ entities_with_label += subwords
+
+ return entities_with_label
+
def group_sub_entities(self, entities: List[dict]) -> dict:
"""
Group together the adjacent tokens with the same entity predicted.
@@ -256,7 +319,6 @@ def group_entities(self, entities: List[dict]) -> List[dict]:
Args:
entities (:obj:`dict`): The entities predicted by the pipeline.
"""
-
entity_groups = []
entity_group_disagg = []
diff --git a/src/transformers/testing_utils.py b/src/transformers/testing_utils.py
index 2292acb66222..4144be2eb9cb 100644
--- a/src/transformers/testing_utils.py
+++ b/src/transformers/testing_utils.py
@@ -389,7 +389,7 @@ def require_torch_tpu(test_case):
def require_torch_gpu(test_case):
- """Decorator marking a test that requires CUDA and PyTorch. """
+ """Decorator marking a test that requires CUDA and PyTorch."""
if torch_device != "cuda":
return unittest.skip("test requires CUDA")(test_case)
else:
@@ -593,14 +593,14 @@ def __repr__(self):
class CaptureStdout(CaptureStd):
- """ Same as CaptureStd but captures only stdout """
+ """Same as CaptureStd but captures only stdout"""
def __init__(self):
super().__init__(err=False)
class CaptureStderr(CaptureStd):
- """ Same as CaptureStd but captures only stderr """
+ """Same as CaptureStd but captures only stderr"""
def __init__(self):
super().__init__(out=False)
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
index fafe8a5597b6..b4e370803ce5 100644
--- a/src/transformers/tokenization_utils.py
+++ b/src/transformers/tokenization_utils.py
@@ -643,7 +643,9 @@ def prepare_for_tokenization(
text (:obj:`str`):
The text to prepare.
is_split_into_words (:obj:`bool`, `optional`, defaults to :obj:`False`):
- Whether or not the text has been pretokenized.
+ Whether or not the input is already pre-tokenized (e.g., split into words). If set to :obj:`True`, the
+ tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
+ which it will tokenize. This is useful for NER or token classification.
kwargs:
Keyword arguments to use for the tokenization.
diff --git a/src/transformers/tokenization_utils_base.py b/src/transformers/tokenization_utils_base.py
index e72b8897677c..2d7f7d851878 100644
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -88,7 +88,7 @@ def __getstate__(self):
@dataclass
class EncodingFast:
- """ This is dummy class because without the `tokenizers` library we don't have these objects anyway """
+ """This is dummy class because without the `tokenizers` library we don't have these objects anyway"""
pass
@@ -1286,8 +1286,9 @@ def all_special_ids(self) -> List[int]:
returned to provide some overlap between truncated and overflowing sequences. The value of this
argument defines the number of overlapping tokens.
is_split_into_words (:obj:`bool`, `optional`, defaults to :obj:`False`):
- Whether or not the input is already pre-tokenized (e.g., split into words), in which case the tokenizer
- will skip the pre-tokenization step. This is useful for NER or token classification.
+ Whether or not the input is already pre-tokenized (e.g., split into words). If set to :obj:`True`, the
+ tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
+ which it will tokenize. This is useful for NER or token classification.
pad_to_multiple_of (:obj:`int`, `optional`):
If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta).
@@ -2278,6 +2279,14 @@ def __call__(
)
if is_batched:
+ if isinstance(text_pair, str):
+ raise TypeError(
+ "when tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`."
+ )
+ if text_pair is not None and len(text) != len(text_pair):
+ raise ValueError(
+ f"batch length of `text`: {len(text)} does not match batch length of `text_pair`: {len(text_pair)}."
+ )
batch_text_or_text_pairs = list(zip(text, text_pair)) if text_pair is not None else text
return self.batch_encode_plus(
batch_text_or_text_pairs=batch_text_or_text_pairs,
diff --git a/src/transformers/tokenization_utils_fast.py b/src/transformers/tokenization_utils_fast.py
index df4dec075841..c62ecdf82a8a 100644
--- a/src/transformers/tokenization_utils_fast.py
+++ b/src/transformers/tokenization_utils_fast.py
@@ -520,7 +520,7 @@ def _save_pretrained(
filename_prefix: Optional[str] = None,
) -> Tuple[str]:
"""
- Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well asin a unique JSON
+ Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON
file containing {config + vocab + added-tokens}.
"""
save_directory = str(save_directory)
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
index d53ad9ac44dc..5565bdb2eab4 100755
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -208,7 +208,7 @@ class Trainer:
Note that if it's a :obj:`torch.utils.data.dataset.IterableDataset` with some randomization and you are
training in a distributed fashion, your iterable dataset should either use a internal attribute
- :obj:`generator` that is a :obj:`torch.Generator` for the randomization that must be identic on all
+ :obj:`generator` that is a :obj:`torch.Generator` for the randomization that must be identical on all
processes (and the Trainer will manually set the seed of this :obj:`generator` at each epoch) or have a
:obj:`set_epoch()` method that internally sets the seed of the RNGs used.
eval_dataset (:obj:`torch.utils.data.dataset.Dataset`, `optional`):
@@ -412,7 +412,12 @@ def __init__(
if args.fp16 and not args.deepspeed: # deepspeed manages its own fp16
if self.fp16_backend == "amp":
self.use_amp = True
- self.scaler = ShardedGradScaler() if self.sharded_ddp is not None else torch.cuda.amp.GradScaler()
+ if is_sagemaker_mp_enabled():
+ self.scaler = smp.amp.GradScaler()
+ elif self.sharded_ddp is not None:
+ self.scaler = ShardedGradScaler()
+ else:
+ self.scaler = torch.cuda.amp.GradScaler()
else:
if not is_apex_available():
raise ImportError(
@@ -420,6 +425,13 @@ def __init__(
)
self.use_apex = True
+ # FP16 + model parallelism in SageMaker: gradient clipping does not work for now so we raise a helpful error.
+ if is_sagemaker_mp_enabled() and self.use_amp and args.max_grad_norm is not None and args.max_grad_norm > 0:
+ raise ValueError(
+ "SageMaker Model Parallelism in mixed precision mode does not support gradient clipping yet. Pass "
+ "along 'max_grad_norm': 0 in your hyperparameters."
+ )
+
# Label smoothing
if self.args.label_smoothing_factor != 0:
self.label_smoother = LabelSmoother(epsilon=self.args.label_smoothing_factor)
@@ -535,6 +547,7 @@ def _get_train_sampler(self) -> Optional[torch.utils.data.sampler.Sampler]:
rank=self.args.process_index,
lengths=lengths,
model_input_name=model_input_name,
+ seed=self.args.seed,
)
else:
@@ -550,10 +563,14 @@ def _get_train_sampler(self) -> Optional[torch.utils.data.sampler.Sampler]:
batch_size=self.args.per_device_train_batch_size,
num_replicas=self.args.world_size,
rank=self.args.process_index,
+ seed=self.args.seed,
)
else:
return DistributedSampler(
- self.train_dataset, num_replicas=self.args.world_size, rank=self.args.process_index
+ self.train_dataset,
+ num_replicas=self.args.world_size,
+ rank=self.args.process_index,
+ seed=self.args.seed,
)
def get_train_dataloader(self) -> DataLoader:
@@ -805,7 +822,7 @@ def num_examples(self, dataloader: DataLoader) -> int:
return len(dataloader.dataset)
def _hp_search_setup(self, trial: Union["optuna.Trial", Dict[str, Any]]):
- """ HP search setup code """
+ """HP search setup code"""
self._trial = trial
if self.hp_search_backend is None or trial is None:
@@ -1607,7 +1624,8 @@ def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor,
inputs = self._prepare_inputs(inputs)
if is_sagemaker_mp_enabled():
- loss_mb = smp_forward_backward(model, inputs, self.args.gradient_accumulation_steps)
+ scaler = self.scaler if self.use_amp else None
+ loss_mb = smp_forward_backward(model, inputs, self.args.gradient_accumulation_steps, scaler=scaler)
return loss_mb.reduce_mean().detach().to(self.args.device)
if self.use_amp:
diff --git a/src/transformers/trainer_callback.py b/src/transformers/trainer_callback.py
index e760ab55c17e..278fb6d0ab5d 100644
--- a/src/transformers/trainer_callback.py
+++ b/src/transformers/trainer_callback.py
@@ -42,7 +42,7 @@ class TrainerState:
In all this class, one step is to be understood as one update step. When using gradient accumulation, one
update step may require several forward and backward passes: if you use :obj:`gradient_accumulation_steps=n`,
- then one update step requires going throuch `n` batches.
+ then one update step requires going through `n` batches.
Args:
epoch (:obj:`float`, `optional`):
@@ -92,14 +92,14 @@ def __post_init__(self):
self.log_history = []
def save_to_json(self, json_path: str):
- """ Save the content of this instance in JSON format inside :obj:`json_path`."""
+ """Save the content of this instance in JSON format inside :obj:`json_path`."""
json_string = json.dumps(dataclasses.asdict(self), indent=2, sort_keys=True) + "\n"
with open(json_path, "w", encoding="utf-8") as f:
f.write(json_string)
@classmethod
def load_from_json(cls, json_path: str):
- """ Create an instance from the content of :obj:`json_path`."""
+ """Create an instance from the content of :obj:`json_path`."""
with open(json_path, "r", encoding="utf-8") as f:
text = f.read()
return cls(**json.loads(text))
@@ -141,15 +141,15 @@ class TrainerControl:
should_log: bool = False
def _new_training(self):
- """ Internal method that resets the variable for a new training. """
+ """Internal method that resets the variable for a new training."""
self.should_training_stop = False
def _new_epoch(self):
- """ Internal method that resets the variable for a new epoch. """
+ """Internal method that resets the variable for a new epoch."""
self.should_epoch_stop = False
def _new_step(self):
- """ Internal method that resets the variable for a new step. """
+ """Internal method that resets the variable for a new step."""
self.should_save = False
self.should_evaluate = False
self.should_log = False
@@ -275,7 +275,7 @@ def on_prediction_step(self, args: TrainingArguments, state: TrainerState, contr
class CallbackHandler(TrainerCallback):
- """ Internal class that just calls the list of callbacks in order. """
+ """Internal class that just calls the list of callbacks in order."""
def __init__(self, callbacks, model, tokenizer, optimizer, lr_scheduler):
self.callbacks = []
diff --git a/src/transformers/trainer_pt_utils.py b/src/transformers/trainer_pt_utils.py
index 0b58904c00fd..5791ac6c3575 100644
--- a/src/transformers/trainer_pt_utils.py
+++ b/src/transformers/trainer_pt_utils.py
@@ -243,7 +243,7 @@ class SequentialDistributedSampler(Sampler):
def __init__(self, dataset, num_replicas=None, rank=None, batch_size=None):
warnings.warn(
- "SequentialDistributedSampler is deprecated and will be removed in v5 of Tranformers.",
+ "SequentialDistributedSampler is deprecated and will be removed in v5 of Transformers.",
FutureWarning,
)
if num_replicas is None:
@@ -294,14 +294,14 @@ def get_tpu_sampler(dataset: torch.utils.data.dataset.Dataset, bach_size: int):
def nested_new_like(arrays, num_samples, padding_index=-100):
- """ Create the same nested structure as `arrays` with a first dimension always at `num_samples`."""
+ """Create the same nested structure as `arrays` with a first dimension always at `num_samples`."""
if isinstance(arrays, (list, tuple)):
return type(arrays)(nested_new_like(x, num_samples) for x in arrays)
return np.full_like(arrays, padding_index, shape=(num_samples, *arrays.shape[1:]))
def expand_like(arrays, new_seq_length, padding_index=-100):
- """ Expand the `arrays` so that the second dimension grows to `new_seq_length`. Uses `padding_index` for padding."""
+ """Expand the `arrays` so that the second dimension grows to `new_seq_length`. Uses `padding_index` for padding."""
result = np.full_like(arrays, padding_index, shape=(arrays.shape[0], new_seq_length) + arrays.shape[2:])
result[:, : arrays.shape[1]] = arrays
return result
@@ -363,7 +363,7 @@ class DistributedTensorGatherer:
def __init__(self, world_size, num_samples, make_multiple_of=None, padding_index=-100):
warnings.warn(
- "DistributedTensorGatherer is deprecated and will be removed in v5 of Tranformers.",
+ "DistributedTensorGatherer is deprecated and will be removed in v5 of Transformers.",
FutureWarning,
)
self.world_size = world_size
@@ -974,10 +974,15 @@ def get_parameter_names(model, forbidden_layer_types):
import smdistributed.modelparallel.torch as smp
@smp.step()
- def smp_forward_backward(model, inputs, gradient_accumulation_steps=1):
- outputs = model(**inputs)
+ def smp_forward_backward(model, inputs, gradient_accumulation_steps=1, scaler=None):
+ with torch.cuda.amp.autocast(enabled=(scaler is not None)):
+ outputs = model(**inputs)
+
loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]
loss /= gradient_accumulation_steps
+ if scaler is not None:
+ loss = scaler.scale(loss).squeeze()
+
model.backward(loss)
return loss
diff --git a/src/transformers/trainer_seq2seq.py b/src/transformers/trainer_seq2seq.py
index 1298c62fc5cf..92d9958fa07f 100644
--- a/src/transformers/trainer_seq2seq.py
+++ b/src/transformers/trainer_seq2seq.py
@@ -199,7 +199,7 @@ def prediction_step(
def _pad_tensors_to_max_len(self, tensor, max_length):
if self.tokenizer is None:
raise ValueError(
- f"Tensor need to be padded to `max_length={max_length}` but no tokenzier was passed when creating "
+ f"Tensor need to be padded to `max_length={max_length}` but no tokenizer was passed when creating "
"this `Trainer`. Make sure to create your `Trainer` with the appropriate tokenizer."
)
# If PAD token is not defined at least EOS token has to be defined
diff --git a/src/transformers/trainer_utils.py b/src/transformers/trainer_utils.py
index 53d2cf7f15a5..7a2bfedf8298 100644
--- a/src/transformers/trainer_utils.py
+++ b/src/transformers/trainer_utils.py
@@ -320,7 +320,7 @@ def __init__(self, skip_memory_metrics=False):
self.init_reported = False
def derive_stage(self):
- """ derives the stage/caller name automatically """
+ """derives the stage/caller name automatically"""
caller = inspect.currentframe().f_back.f_back.f_code.co_name
if caller in self.stages:
return self.stages[caller]
@@ -330,7 +330,7 @@ def derive_stage(self):
)
def cpu_mem_used(self):
- """ get resident set size memory for the current process """
+ """get resident set size memory for the current process"""
return self.process.memory_info().rss
def peak_monitor_func(self):
@@ -346,7 +346,7 @@ def peak_monitor_func(self):
break
def start(self):
- """ start tracking for the caller's stage """
+ """start tracking for the caller's stage"""
if self.skip_memory_metrics:
return
@@ -376,7 +376,7 @@ def start(self):
peak_monitor_thread.start()
def stop(self, stage):
- """ stop tracking for the passed stage """
+ """stop tracking for the passed stage"""
# deal with nested calls of eval during train - simply ignore those
if self.cur_stage is not None and self.cur_stage != stage:
@@ -416,7 +416,7 @@ def stop(self, stage):
self.cur_stage = None
def update_metrics(self, stage, metrics):
- """ stop tracking for the passed stage """
+ """stop tracking for the passed stage"""
if self.skip_memory_metrics:
return
@@ -438,7 +438,7 @@ def update_metrics(self, stage, metrics):
metrics[f"{stage}_mem_gpu_{t}_delta"] = self.gpu[stage][t]
def stop_and_update_metrics(self, metrics=None):
- """ combine stop + update in one call for simpler code """
+ """combine stop + update in one call for simpler code"""
if self.skip_memory_metrics:
return
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
index a91921d466a5..30e433cfc7c6 100644
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -70,9 +70,6 @@ class TrainingArguments:
`__ arguments that can be specified on the command
line.
-
-
-
Parameters:
output_dir (:obj:`str`):
The output directory where the model predictions and checkpoints will be written.
@@ -625,6 +622,14 @@ def __post_init__(self):
elif ShardedDDPOption.ZERO_DP_2 in self.sharded_ddp and ShardedDDPOption.ZERO_DP_3 in self.sharded_ddp:
raise ValueError("`--sharded_ddp zero_dp_2` is not compatible with `--sharded_ddp zero_dp_3`.")
+ if self.deepspeed:
+ # - must be run very last in arg parsing, since it will use a lot of these settings.
+ # - must be run before the model is created.
+ from transformers.integrations import DeepSpeedConfigHF
+
+ # will be used later by the Trainer (leave self.deepspeed unmodified in case a user relies on it not to be modified)
+ self.deepspeed_config_hf = DeepSpeedConfigHF(self)
+
def __repr__(self):
# We override the default repr to remove deprecated arguments from the repr. This method should be removed once
# those deprecated arguments are removed form TrainingArguments. (TODO: v5)
diff --git a/src/transformers/utils/logging.py b/src/transformers/utils/logging.py
index 256343221a6a..8e7b592cf055 100644
--- a/src/transformers/utils/logging.py
+++ b/src/transformers/utils/logging.py
@@ -140,7 +140,7 @@ def get_verbosity() -> int:
def set_verbosity(verbosity: int) -> None:
"""
- Set the vebosity level for the 🤗 Transformers's root logger.
+ Set the verbosity level for the 🤗 Transformers's root logger.
Args:
verbosity (:obj:`int`):
diff --git a/src/transformers/utils/versions.py b/src/transformers/utils/versions.py
index 73151487bc71..36125d868117 100644
--- a/src/transformers/utils/versions.py
+++ b/src/transformers/utils/versions.py
@@ -115,12 +115,12 @@ def require_version(requirement: str, hint: Optional[str] = None) -> None:
def require_version_core(requirement):
- """ require_version wrapper which emits a core-specific hint on failure """
+ """require_version wrapper which emits a core-specific hint on failure"""
hint = "Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master"
return require_version(requirement, hint)
def require_version_examples(requirement):
- """ require_version wrapper which emits examples-specific hint on failure """
+ """require_version wrapper which emits examples-specific hint on failure"""
hint = "Try: pip install -r examples/requirements.txt"
return require_version(requirement, hint)
diff --git a/templates/adding_a_new_example_script/{{cookiecutter.directory_name}}/run_{{cookiecutter.example_shortcut}}.py b/templates/adding_a_new_example_script/{{cookiecutter.directory_name}}/run_{{cookiecutter.example_shortcut}}.py
index d40328b925cf..48590fe16712 100755
--- a/templates/adding_a_new_example_script/{{cookiecutter.directory_name}}/run_{{cookiecutter.example_shortcut}}.py
+++ b/templates/adding_a_new_example_script/{{cookiecutter.directory_name}}/run_{{cookiecutter.example_shortcut}}.py
@@ -157,17 +157,17 @@ class DataTrainingArguments:
"value if set."
},
)
- max_val_samples: Optional[int] = field(
+ max_eval_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
},
)
- max_test_samples: Optional[int] = field(
+ max_predict_samples: Optional[int] = field(
default=None,
metadata={
- "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
+ "help": "For debugging purposes or quicker training, truncate the number of prediction examples to this "
"value if set."
},
)
@@ -379,8 +379,8 @@ def tokenize_function(examples):
raise ValueError("--do_eval requires a validation dataset")
eval_dataset = datasets["validation"]
# Selecting samples from dataset
- if data_args.max_val_samples is not None:
- eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
+ if data_args.max_eval_samples is not None:
+ eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
# tokenize validation dataset
eval_dataset = eval_dataset.map(
tokenize_function,
@@ -393,12 +393,12 @@ def tokenize_function(examples):
if training_args.do_predict:
if "test" not in datasets:
raise ValueError("--do_predict requires a test dataset")
- test_dataset = datasets["test"]
+ predict_dataset = datasets["test"]
# Selecting samples from dataset
- if data_args.max_test_samples is not None:
- test_dataset = test_dataset.select(range(data_args.max_test_samples))
- # tokenize test dataset
- test_dataset = test_dataset.map(
+ if data_args.max_predict_samples is not None:
+ predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
+ # tokenize predict dataset
+ predict_dataset = predict_dataset.map(
tokenize_function,
batched=True,
num_proc=data_args.preprocessing_num_workers,
@@ -455,8 +455,8 @@ def tokenize_function(examples):
metrics = trainer.evaluate()
- max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
- metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))
+ max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+ metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
@@ -464,13 +464,13 @@ def tokenize_function(examples):
# Prediction
if training_args.do_predict:
logger.info("*** Predict ***")
- predictions, labels, metrics = trainer.predict(test_dataset)
+ predictions, labels, metrics = trainer.predict(predict_dataset)
- max_test_samples = data_args.max_test_samples if data_args.max_test_samples is not None else len(test_dataset)
- metrics["test_samples"] = min(max_test_samples, len(test_dataset))
+ max_predict_samples = data_args.max_predict_samples if data_args.max_predict_samples is not None else len(predict_dataset)
+ metrics["predict_samples"] = min(max_predict_samples, len(predict_dataset))
- trainer.log_metrics("test", metrics)
- trainer.save_metrics("test", metrics)
+ trainer.log_metrics("predict", metrics)
+ trainer.save_metrics("predict", metrics)
# write custom code for saving predictions according to task
diff --git a/templates/adding_a_new_model/cookiecutter-template-{{cookiecutter.modelname}}/modeling_tf_{{cookiecutter.lowercase_modelname}}.py b/templates/adding_a_new_model/cookiecutter-template-{{cookiecutter.modelname}}/modeling_tf_{{cookiecutter.lowercase_modelname}}.py
index 6b04672db161..d4214787dc7b 100644
--- a/templates/adding_a_new_model/cookiecutter-template-{{cookiecutter.modelname}}/modeling_tf_{{cookiecutter.lowercase_modelname}}.py
+++ b/templates/adding_a_new_model/cookiecutter-template-{{cookiecutter.modelname}}/modeling_tf_{{cookiecutter.lowercase_modelname}}.py
@@ -147,7 +147,6 @@ def call(
return final_embeddings
-
# Copied from transformers.models.bert.modeling_tf_bert.TFBertSelfAttention with Bert->{{cookiecutter.camelcase_modelname}}
class TF{{cookiecutter.camelcase_modelname}}SelfAttention(tf.keras.layers.Layer):
def __init__(self, config: {{cookiecutter.camelcase_modelname}}Config, **kwargs):
@@ -352,6 +351,7 @@ def call(
return outputs
+
# Copied from transformers.models.bert.modeling_tf_bert.TFBertEncoder with Bert->{{cookiecutter.camelcase_modelname}}
class TF{{cookiecutter.camelcase_modelname}}Encoder(tf.keras.layers.Layer):
def __init__(self, config: {{cookiecutter.camelcase_modelname}}Config, **kwargs):
@@ -625,7 +625,6 @@ class TF{{cookiecutter.camelcase_modelname}}PreTrainedModel(TFPreTrainedModel):
base_model_prefix = "{{cookiecutter.lowercase_modelname}}"
-
{{cookiecutter.uppercase_modelname}}_START_DOCSTRING = r"""
This model inherits from :class:`~transformers.TFPreTrainedModel`. Check the superclass documentation for the
@@ -885,6 +884,7 @@ def serving_output(self, output: TFMaskedLMOutput) -> TFMaskedLMOutput:
return TFMaskedLMOutput(logits=output.logits, hidden_states=hs, attentions=attns)
+
@add_start_docstrings(
"""{{cookiecutter.modelname}} Model with a `language modeling` head on top for CLM fine-tuning. """, {{cookiecutter.uppercase_modelname}}_START_DOCSTRING
)
@@ -1728,16 +1728,18 @@ def __init__(self, config: {{cookiecutter.camelcase_modelname}}Config, **kwargs)
self.fc2 = tf.keras.layers.Dense(self.embed_dim, name="fc2")
self.final_layer_norm = tf.keras.layers.LayerNormalization(epsilon=1e-5, name="final_layer_norm")
- def call(self, hidden_states: tf.Tensor, attention_mask: tf.Tensor, training=False):
+ def call(self, hidden_states: tf.Tensor, attention_mask: tf.Tensor, layer_head_mask: tf.Tensor, training=False):
"""
Args:
hidden_states (:obj:`tf.Tensor`): input to the layer of shape `(seq_len, batch, embed_dim)`
attention_mask (:obj:`tf.Tensor`): attention mask of size
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
+ layer_head_mask (:obj:`tf.Tensor`): mask for attention heads in a given layer of size
+ `(encoder_attention_heads,)`
"""
residual = hidden_states
hidden_states, self_attn_weights, _ = self.self_attn(
- hidden_states=hidden_states, attention_mask=attention_mask
+ hidden_states=hidden_states, attention_mask=attention_mask, layer_head_mask=layer_head_mask
)
# The tf.debugging asserts are not compliant with XLA then they
@@ -1798,6 +1800,8 @@ def call(
attention_mask: Optional[tf.Tensor] = None,
encoder_hidden_states: Optional[tf.Tensor] = None,
encoder_attention_mask: Optional[tf.Tensor] = None,
+ layer_head_mask: Optional[tf.Tensor] = None,
+ cross_attn_layer_head_mask: Optional[tf.Tensor] = None,
past_key_value: Optional[Tuple[tf.Tensor]] = None,
training=False,
) -> Tuple[tf.Tensor, tf.Tensor, Tuple[Tuple[tf.Tensor]]]:
@@ -1809,6 +1813,10 @@ def call(
encoder_hidden_states (:obj:`tf.Tensor`): cross attention input to the layer of shape `(seq_len, batch, embed_dim)`
encoder_attention_mask (:obj:`tf.Tensor`): encoder attention mask of size
`(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.
+ layer_head_mask (:obj:`tf.Tensor`): mask for attention heads in a given layer of size
+ `(decoder_attention_heads,)`
+ cross_attn_layer_head_mask (:obj:`tf.Tensor`): mask for heads of the cross-attention module.
+ `(decoder_attention_heads,)`
past_key_value (:obj:`Tuple(tf.Tensor)`): cached past key and value projection states
"""
residual = hidden_states
@@ -1821,6 +1829,7 @@ def call(
hidden_states=hidden_states,
past_key_value=self_attn_past_key_value,
attention_mask=attention_mask,
+ layer_head_mask=layer_head_mask,
)
hidden_states = self.dropout(hidden_states, training=training)
hidden_states = residual + hidden_states
@@ -1828,15 +1837,17 @@ def call(
# Cross-Attention Block
cross_attn_present_key_value = None
+ cross_attn_weights = None
if encoder_hidden_states is not None:
residual = hidden_states
# cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
- hidden_states, _, cross_attn_present_key_value = self.encoder_attn(
+ hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
hidden_states=hidden_states,
key_value_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
+ layer_head_mask=cross_attn_layer_head_mask,
past_key_value=cross_attn_past_key_value,
)
hidden_states = self.dropout(hidden_states, training=training)
@@ -1858,6 +1869,7 @@ def call(
return (
hidden_states,
self_attn_weights,
+ cross_attn_layer_head_mask,
present_key_value,
)
@@ -1965,6 +1977,24 @@ def serving(self, inputs):
the right for denoising pre-training following the paper.
decoder_attention_mask (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`):
will be made by default and ignore pad tokens. It is not recommended to set this for most use cases.
+ head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
+ decoder_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
encoder_outputs (:obj:`tf.FloatTensor`, `optional`):
hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.
of shape :obj:`(batch_size, sequence_length, hidden_size)` is a sequence of
@@ -2013,7 +2043,6 @@ def __init__(self, config: {{cookiecutter.camelcase_modelname}}Config, embed_tok
self.max_source_positions = config.max_position_embeddings
self.embed_scale = tf.math.sqrt(float(config.d_model)) if config.scale_embedding else 1.0
-
self.embed_tokens = embed_tokens
self.embed_positions = TF{{cookiecutter.camelcase_modelname}}LearnedPositionalEmbedding(
config.max_position_embeddings,
@@ -2034,6 +2063,7 @@ def call(
input_ids=None,
inputs_embeds=None,
attention_mask=None,
+ head_mask=None,
output_attentions=None,
output_hidden_states=None,
return_dict=None,
@@ -2058,6 +2088,12 @@ def call(
- 0 for tokens that are **masked**.
`What are attention masks? <../glossary.html#attention-mask>`__
+ head_mask (:obj:`tf.Tensor` of shape :obj:`(encoder_layers, encoder_attention_heads)`, `optional):
+ Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
inputs_embeds (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
Optionally, instead of passing :obj:`input_ids` you can choose to directly pass an embedded
representation. This is useful if you want more control over how to convert :obj:`input_ids` indices
@@ -2082,6 +2118,7 @@ def call(
config=self.config,
input_ids=input_ids,
attention_mask=attention_mask,
+ head_mask=head_mask,
inputs_embeds=inputs_embeds,
output_attentions=output_attentions,
output_hidden_states=output_hidden_states,
@@ -2115,6 +2152,16 @@ def call(
encoder_states = () if inputs["output_hidden_states"] else None
all_attentions = () if inputs["output_attentions"] else None
+ # check if head_mask has a correct number of layers specified if desired
+ # The tf.debugging asserts are not compliant with XLA then they
+ # have to be disabled in other modes than eager.
+ if inputs["head_mask"] is not None and tf.executing_eagerly():
+ tf.debugging.assert_equal(
+ shape_list(inputs["head_mask"])[0],
+ len(self.layers),
+ message=f"The head_mask should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs['head_mask'])[0]}.",
+ )
+
# encoder layers
for encoder_layer in self.layers:
@@ -2125,7 +2172,11 @@ def call(
if inputs["training"] and (dropout_probability < self.layerdrop): # skip the layer
continue
- hidden_states, attn = encoder_layer(hidden_states, inputs["attention_mask"])
+ hidden_states, attn = encoder_layer(
+ hidden_states,
+ inputs["attention_mask"],
+ inputs["head_mask"][idx] if inputs["head_mask"] is not None else None,
+ )
if inputs["output_attentions"]:
all_attentions += (attn,)
@@ -2181,6 +2232,8 @@ def call(
attention_mask=None,
encoder_hidden_states=None,
encoder_attention_mask=None,
+ head_mask=None,
+ cross_attn_head_mask=None,
past_key_values=None,
use_cache=None,
output_attentions=None,
@@ -2218,6 +2271,18 @@ def call(
- 0 for tokens that are **masked**.
`What are attention masks? <../glossary.html#attention-mask>`__
+ head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
+ cross_attn_head_mask (:obj:`tf.Tensor` of shape :obj:`(decoder_layers, decoder_attention_heads)`, `optional`):
+ Mask to nullify selected heads of the cross-attention modules. Mask values selected in ``[0, 1]``:
+
+ - 1 indicates the head is **not masked**,
+ - 0 indicates the head is **masked**.
+
past_key_values (:obj:`Tuple[Tuple[tf.Tensor]]` of length :obj:`config.n_layers` with each tuple having 2 tuples each of which has 2 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up
decoding.
@@ -2252,6 +2317,8 @@ def call(
attention_mask=attention_mask,
encoder_hidden_states=encoder_hidden_states,
encoder_attention_mask=encoder_attention_mask,
+ head_mask=head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
inputs_embeds=inputs_embeds,
past_key_values=past_key_values,
use_cache=use_cache,
@@ -2297,8 +2364,20 @@ def call(
# decoder layers
all_hidden_states = () if inputs["output_hidden_states"] else None
all_self_attns = () if inputs["output_attentions"] else None
+ all_cross_attns = () if (inputs["output_attentions"] and inputs["encoder_hidden_states"] is not None) else None
present_key_values = () if inputs["use_cache"] else None
+ # check if head_mask and cross_attn_head_mask have a correct number of layers specified if desired
+ # The tf.debugging asserts are not compliant with XLA then they
+ # have to be disabled in other modes than eager.
+ for attn_mask in ["head_mask", "cross_attn_head_mask"]:
+ if inputs[attn_mask] is not None and tf.executing_eagerly():
+ tf.debugging.assert_equal(
+ shape_list(inputs[attn_mask])[0],
+ len(self.layers),
+ message=f"The {attn_mask} should be specified for {len(self.layers)} layers, but it is for {shape_list(inputs[attn_mask])[0]}.",
+ )
+
for idx, decoder_layer in enumerate(self.layers):
# add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)
if inputs["output_hidden_states"]:
@@ -2311,11 +2390,15 @@ def call(
past_key_value = inputs["past_key_values"][idx] if inputs["past_key_values"] is not None else None
- hidden_states, layer_self_attn, present_key_value = decoder_layer(
+ hidden_states, layer_self_attn, layer_cross_attn, present_key_value = decoder_layer(
hidden_states,
attention_mask=combined_attention_mask,
encoder_hidden_states=inputs["encoder_hidden_states"],
encoder_attention_mask=inputs["encoder_attention_mask"],
+ layer_head_mask=inputs["head_mask"][idx] if inputs["head_mask"] is not None else None,
+ cross_attn_layer_head_mask=inputs["cross_attn_head_mask"][idx]
+ if inputs["cross_attn_head_mask"] is not None
+ else None,
past_key_value=past_key_value,
)
@@ -2325,23 +2408,30 @@ def call(
if inputs["output_attentions"]:
all_self_attns += (layer_self_attn,)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns += (layer_cross_attn,)
+
if inputs["output_hidden_states"]:
all_hidden_states += (hidden_states,)
if inputs["output_attentions"]:
all_self_attns = list(all_self_attns)
+ if inputs["encoder_hidden_states"] is not None:
+ all_cross_attns = list(all_cross_attns)
+
if inputs["use_cache"]:
present_key_values = (inputs["encoder_hidden_states"], present_key_values)
if not inputs["return_dict"]:
- return hidden_states, present_key_values, all_hidden_states, all_self_attns
+ return hidden_states, present_key_values, all_hidden_states, all_self_attns, all_cross_attns
else:
- return TFBaseModelOutputWithPast(
+ return TFBaseModelOutputWithPastAndCrossAttentions(
last_hidden_state=hidden_states,
past_key_values=present_key_values,
hidden_states=all_hidden_states,
attentions=all_self_attns,
+ cross_attentions=all_cross_attns,
)
@tf.function
@@ -2413,6 +2503,9 @@ def call(
attention_mask=None,
decoder_input_ids=None,
decoder_attention_mask=None,
+ head_mask=None,
+ decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -2431,6 +2524,9 @@ def call(
attention_mask=attention_mask,
decoder_input_ids=decoder_input_ids,
decoder_attention_mask=decoder_attention_mask,
+ head_mask=head_mask,
+ decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -2450,6 +2546,7 @@ def call(
inputs["encoder_outputs"] = self.encoder(
input_ids=inputs["input_ids"],
attention_mask=inputs["attention_mask"],
+ head_mask=inputs["head_mask"],
inputs_embeds=inputs["inputs_embeds"],
output_attentions=inputs["output_attentions"],
output_hidden_states=inputs["output_hidden_states"],
@@ -2472,6 +2569,8 @@ def call(
attention_mask=inputs["decoder_attention_mask"],
encoder_hidden_states=inputs["encoder_outputs"][0],
encoder_attention_mask=inputs["attention_mask"],
+ head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["decoder_inputs_embeds"],
use_cache=inputs["use_cache"],
@@ -2489,6 +2588,7 @@ def call(
past_key_values=decoder_outputs.past_key_values,
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
+ cross_attentions=decoder_outputs.cross_attentions,
encoder_last_hidden_state=inputs["encoder_outputs"].last_hidden_state,
encoder_hidden_states=inputs["encoder_outputs"].hidden_states,
encoder_attentions=inputs["encoder_outputs"].attentions,
@@ -2524,6 +2624,9 @@ def call(
attention_mask=None,
decoder_input_ids=None,
decoder_attention_mask=None,
+ head_mask=None,
+ decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[Union[Tuple, TFBaseModelOutput]] = None,
past_key_values=None,
inputs_embeds=None,
@@ -2542,6 +2645,9 @@ def call(
attention_mask=attention_mask,
decoder_input_ids=decoder_input_ids,
decoder_attention_mask=decoder_attention_mask,
+ head_mask=head_mask,
+ decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -2559,6 +2665,9 @@ def call(
attention_mask=inputs["attention_mask"],
decoder_input_ids=inputs["decoder_input_ids"],
decoder_attention_mask=inputs["decoder_attention_mask"],
+ head_mask=inputs["head_mask"],
+ decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
encoder_outputs=inputs["encoder_outputs"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
@@ -2577,6 +2686,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -2585,6 +2695,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
@@ -2637,6 +2748,9 @@ def call(
attention_mask=None,
decoder_input_ids=None,
decoder_attention_mask=None,
+ head_mask=None,
+ decoder_head_mask=None,
+ cross_attn_head_mask=None,
encoder_outputs: Optional[TFBaseModelOutput] = None,
past_key_values=None,
inputs_embeds=None,
@@ -2672,6 +2786,9 @@ def call(
attention_mask=attention_mask,
decoder_input_ids=decoder_input_ids,
decoder_attention_mask=decoder_attention_mask,
+ head_mask=head_mask,
+ decoder_head_mask=decoder_head_mask,
+ cross_attn_head_mask=cross_attn_head_mask,
encoder_outputs=encoder_outputs,
past_key_values=past_key_values,
inputs_embeds=inputs_embeds,
@@ -2698,6 +2815,9 @@ def call(
decoder_input_ids=inputs["decoder_input_ids"],
encoder_outputs=inputs["encoder_outputs"],
decoder_attention_mask=inputs["decoder_attention_mask"],
+ head_mask=inputs["head_mask"],
+ decoder_head_mask=inputs["decoder_head_mask"],
+ cross_attn_head_mask=inputs["cross_attn_head_mask"],
past_key_values=inputs["past_key_values"],
inputs_embeds=inputs["inputs_embeds"],
decoder_inputs_embeds=inputs["decoder_inputs_embeds"],
@@ -2720,6 +2840,7 @@ def call(
past_key_values=outputs.past_key_values, # index 1 of d outputs
decoder_hidden_states=outputs.decoder_hidden_states, # index 2 of d outputs
decoder_attentions=outputs.decoder_attentions, # index 3 of d outputs
+ cross_attentions=outputs.cross_attentions, # index 4 of d outputs
encoder_last_hidden_state=outputs.encoder_last_hidden_state, # index 0 of encoder outputs
encoder_hidden_states=outputs.encoder_hidden_states, # 1 of e out
encoder_attentions=outputs.encoder_attentions, # 2 of e out
@@ -2730,6 +2851,7 @@ def serving_output(self, output):
pkv = tf.tuple(output.past_key_values)[1] if self.config.use_cache else None
dec_hs = tf.convert_to_tensor(output.decoder_hidden_states) if self.config.output_hidden_states else None
dec_attns = tf.convert_to_tensor(output.decoder_attentions) if self.config.output_attentions else None
+ cross_attns = tf.convert_to_tensor(output.cross_attentions) if self.config.output_attentions else None
enc_hs = tf.convert_to_tensor(output.encoder_hidden_states) if self.config.output_hidden_states else None
enc_attns = tf.convert_to_tensor(output.encoder_attentions) if self.config.output_attentions else None
@@ -2738,6 +2860,7 @@ def serving_output(self, output):
past_key_values=pkv,
decoder_hidden_states=dec_hs,
decoder_attentions=dec_attns,
+ cross_attentions=cross_attns,
encoder_last_hidden_state=output.encoder_last_hidden_state,
encoder_hidden_states=enc_hs,
encoder_attentions=enc_attns,
diff --git a/tests/deepspeed/ds_config_zero2.json b/tests/deepspeed/ds_config_zero2.json
index a516f33125ef..ef180edd1e5b 100644
--- a/tests/deepspeed/ds_config_zero2.json
+++ b/tests/deepspeed/ds_config_zero2.json
@@ -1,6 +1,6 @@
{
"fp16": {
- "enabled": true,
+ "enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
@@ -8,36 +8,40 @@
"min_loss_scale": 1
},
- "zero_optimization": {
- "stage": 2,
- "allgather_partitions": true,
- "allgather_bucket_size": 2e8,
- "overlap_comm": true,
- "reduce_scatter": true,
- "reduce_bucket_size": 2e8,
- "contiguous_gradients": true,
- "cpu_offload": true
- },
-
"optimizer": {
"type": "AdamW",
"params": {
- "lr": 3e-5,
- "betas": [0.8, 0.999],
- "eps": 1e-8,
- "weight_decay": 3e-7
+ "lr": "auto",
+ "betas": "auto",
+ "eps": "auto",
+ "weight_decay": "auto"
}
},
"scheduler": {
"type": "WarmupLR",
"params": {
- "warmup_min_lr": 0,
- "warmup_max_lr": 3e-5,
- "warmup_num_steps": 500
+ "warmup_min_lr": "auto",
+ "warmup_max_lr": "auto",
+ "warmup_num_steps": "auto"
}
},
+ "zero_optimization": {
+ "stage": 2,
+ "allgather_partitions": true,
+ "allgather_bucket_size": 2e8,
+ "overlap_comm": true,
+ "reduce_scatter": true,
+ "reduce_bucket_size": 2e8,
+ "contiguous_gradients": true,
+ "cpu_offload": true
+ },
+
+ "gradient_accumulation_steps": "auto",
+ "gradient_clipping": "auto",
"steps_per_print": 2000,
+ "train_batch_size": "auto",
+ "train_micro_batch_size_per_gpu": "auto",
"wall_clock_breakdown": false
}
diff --git a/tests/deepspeed/ds_config_zero3.json b/tests/deepspeed/ds_config_zero3.json
index 0f909959521e..6f7a80e9e455 100644
--- a/tests/deepspeed/ds_config_zero3.json
+++ b/tests/deepspeed/ds_config_zero3.json
@@ -1,6 +1,6 @@
{
"fp16": {
- "enabled": true,
+ "enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
@@ -8,41 +8,50 @@
"min_loss_scale": 1
},
- "zero_optimization": {
- "stage": 3,
- "cpu_offload": true,
- "cpu_offload_params": true,
- "cpu_offload_use_pin_memory" : true,
- "overlap_comm": true,
- "contiguous_gradients": true,
- "sub_group_size": 1e14,
- "reduce_bucket_size": 0,
- "stage3_prefetch_bucket_size": 0,
- "stage3_param_persistence_threshold": 0,
- "stage3_max_live_parameters": 1e9,
- "stage3_max_reuse_distance": 1e9,
- "stage3_gather_fp16_weights_on_model_save": true
- },
-
"optimizer": {
"type": "AdamW",
"params": {
- "lr": 3e-5,
- "betas": [0.8, 0.999],
- "eps": 1e-8,
- "weight_decay": 3e-7
+ "lr": "auto",
+ "betas": "auto",
+ "eps": "auto",
+ "weight_decay": "auto"
}
},
"scheduler": {
"type": "WarmupLR",
"params": {
- "warmup_min_lr": 0,
- "warmup_max_lr": 3e-5,
- "warmup_num_steps": 500
+ "warmup_min_lr": "auto",
+ "warmup_max_lr": "auto",
+ "warmup_num_steps": "auto"
}
},
+ "zero_optimization": {
+ "stage": 3,
+ "offload_optimizer": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "offload_param": {
+ "device": "cpu",
+ "pin_memory": true
+ },
+ "overlap_comm": true,
+ "contiguous_gradients": true,
+ "sub_group_size": 1e14,
+ "reduce_bucket_size": "auto",
+ "stage3_prefetch_bucket_size": "auto",
+ "stage3_param_persistence_threshold": "auto",
+ "stage3_max_live_parameters": 1e9,
+ "stage3_max_reuse_distance": 1e9,
+ "stage3_gather_fp16_weights_on_model_save": true
+ },
+
+ "gradient_accumulation_steps": "auto",
+ "gradient_clipping": "auto",
"steps_per_print": 2000,
+ "train_batch_size": "auto",
+ "train_micro_batch_size_per_gpu": "auto",
"wall_clock_breakdown": false
}
diff --git a/tests/deepspeed/test_deepspeed.py b/tests/deepspeed/test_deepspeed.py
index 9868966a5a32..52f9bd72f121 100644
--- a/tests/deepspeed/test_deepspeed.py
+++ b/tests/deepspeed/test_deepspeed.py
@@ -42,7 +42,7 @@
from test_trainer import TrainerIntegrationCommon # noqa
if is_torch_available():
- from test_trainer import get_regression_trainer # noqa
+ from test_trainer import RegressionModelConfig, RegressionPreTrainedModel, get_regression_trainer # noqa
set_seed(42)
@@ -66,6 +66,10 @@ def require_deepspeed(test_case):
return test_case
+if is_deepspeed_available():
+ from deepspeed.utils import logger as deepspeed_logger # noqa
+ from transformers.integrations import deepspeed_config, is_deepspeed_zero3_enabled # noqa
+
ZERO2 = "zero2"
ZERO3 = "zero3"
stages = [ZERO2, ZERO3]
@@ -115,14 +119,8 @@ def setUp(self):
with io.open(self.ds_config_file[ZERO3], "r", encoding="utf-8") as f:
self.ds_config_dict[ZERO3] = json.load(f)
- def tearDown(self):
- # XXX: Fixme - this is a temporary band-aid since this global variable impacts other tests
- import transformers
-
- transformers.integrations._is_deepspeed_zero3_enabled = None
-
def get_config_dict(self, stage):
- """ As the tests modify the dict, always make a copy """
+ """As the tests modify the dict, always make a copy"""
config = deepcopy(self.ds_config_dict[stage])
if stage == ZERO3:
# This setting slows things down, so don't enable it by default unless needed by a test.
@@ -173,25 +171,65 @@ def test_hf_scheduler_ds_optimizer(self):
trainer = get_regression_trainer(local_rank=0, deepspeed=ds_config_zero2_dict)
with self.assertRaises(Exception) as context:
trainer.train()
- self.assertTrue("HF scheduler + DeepSpeed optimizer combination is not possible" in str(context.exception))
+ self.assertTrue(
+ "HF scheduler + DeepSpeed optimizer combination is not possible" in str(context.exception),
+ f"got exception: {context.exception}",
+ )
+
+ def test_stage3_nvme_offload(self):
+ with mockenv_context(**self.dist_env_1_gpu):
+ # this actually doesn't have to be on NVMe, any storage will do since this test only
+ # runs a simple check that we can use some directory as if it were NVMe
+ nvme_path = self.get_auto_remove_tmp_dir()
+ nvme_config = dict(device="nvme", nvme_path=nvme_path)
+ ds_config_zero3_dict = self.get_config_dict(ZERO3)
+ ds_config_zero3_dict["zero_optimization"]["offload_optimizer"] = nvme_config
+ ds_config_zero3_dict["zero_optimization"]["offload_param"] = nvme_config
+ trainer = get_regression_trainer(local_rank=0, deepspeed=ds_config_zero3_dict)
+ with CaptureLogger(deepspeed_logger) as cs:
+ trainer.train()
+ self.assertIn("DeepSpeed info", cs.out, "expected DeepSpeed logger output but got none")
+
+ # --- These tests need to run on both zero stages --- #
+
+ @parameterized.expand(stages)
+ def test_fp32(self, stage):
+ ds_config_dict = self.get_config_dict(stage)
+ ds_config_dict["fp16"]["enabled"] = False # force non-fp16 mode
+
+ # XXX: do we go via from_pretrained in zero 3 here? need to test zero.Init(dtype=torch.float)
+
+ # XXX: rewrite this test once fp32 is supported by DeepSpeed
+ with mockenv_context(**self.dist_env_1_gpu):
+ trainer = get_regression_trainer(local_rank=0, deepspeed=ds_config_dict)
+ with self.assertRaises(Exception) as context:
+ trainer.train()
+ self.assertIn(
+ "ZeRO is only supported if fp16 is enabled",
+ str(context.exception),
+ f"got exception: {context.exception}",
+ )
- def test_hf_optimizer_with_offload(self):
+ @parameterized.expand(stages)
+ def test_hf_optimizer_with_offload(self, stage):
# must not allow non-DS optimizer when using ZERO-offload
+ ds_config_dict = self.get_config_dict(stage)
+ del ds_config_dict["optimizer"] # force default HF Trainer optimizer
+ # force cpu offload
+ if stage == "stage2":
+ ds_config_dict["zero_optimization"]["cpu_offload"] = True
+ elif stage == "stage3":
+ ds_config_dict["zero_optimization"]["offload_optimizer"]["device"] = "cpu"
with mockenv_context(**self.dist_env_1_gpu):
- ds_config_zero2_dict = self.get_config_dict(ZERO2)
- del ds_config_zero2_dict["optimizer"] # force default HF Trainer optimizer
- ds_config_zero2_dict["zero_optimization"]["cpu_offload"] = True
- # sanity check - should the default config change
- assert (
- "cpu_offload" in ds_config_zero2_dict["zero_optimization"]
- and ds_config_zero2_dict["zero_optimization"]["cpu_offload"] is True
- ), "ensure the config is set up correctly"
- trainer = get_regression_trainer(local_rank=0, deepspeed=ds_config_zero2_dict)
+ trainer = get_regression_trainer(local_rank=0, deepspeed=ds_config_dict)
with self.assertRaises(Exception) as context:
trainer.train()
- self.assertTrue("ZeRO Offload can only work with DeepSpeed optimizers" in str(context.exception))
+ self.assertIn(
+ "ZeRO Offload can only work with DeepSpeed optimizers",
+ str(context.exception),
+ f"got exception: {context.exception}",
+ )
- # --- These tests need to run on both zero stages --- #
@parameterized.expand(stages)
def test_fake_notebook_no_launcher(self, stage):
# this setup emulates a notebook where a launcher needs to be emulated by hand
@@ -199,14 +237,12 @@ def test_fake_notebook_no_launcher(self, stage):
# note that unittest resets sys.stdout each test, so `CaptureStd` will work here to capture
# DeepSpeed log if this test happens to run first in this pytest worker. But it will fail if
# it's run not as a first test as `sys.stdout` will no longer be the same. So we either have
- # to reset `logger.handlers[0].setStream(sys.stdout)` or directly capture from the logger.
- from deepspeed.utils import logger
-
- with CaptureLogger(logger) as cs:
- with mockenv_context(**self.dist_env_1_gpu):
- trainer = get_regression_trainer(local_rank=0, deepspeed=self.ds_config_file[stage])
+ # to reset `deepspeed_logger.handlers[0].setStream(sys.stdout)` or directly capture from the deepspeed_logger.
+ with mockenv_context(**self.dist_env_1_gpu):
+ trainer = get_regression_trainer(local_rank=0, deepspeed=self.ds_config_file[stage])
+ with CaptureLogger(deepspeed_logger) as cs:
trainer.train()
- assert "DeepSpeed info" in cs.out, "expected DeepSpeed logger output but got none"
+ self.assertIn("DeepSpeed info", cs.out, "expected DeepSpeed logger output but got none")
@parameterized.expand(stages)
def test_early_get_last_lr(self, stage):
@@ -425,12 +461,44 @@ def test_can_resume_training_normal(self, stage):
self.assertEqual(b, b1)
self.check_trainer_state_are_the_same(state, state1)
+ def test_config_object(self):
+ # test that we can switch from zero2 to zero3 in the same process for example
+ # test is_zero, etc.
+ output_dir = self.get_auto_remove_tmp_dir()
+ kwargs = dict(output_dir=output_dir, train_len=8)
+
+ with mockenv_context(**self.dist_env_1_gpu):
+ ds_config_zero3_dict = self.get_config_dict("zero3")
+ ds_config_zero2_dict = self.get_config_dict("zero2")
+
+ trainer = get_regression_trainer(deepspeed=ds_config_zero3_dict, **kwargs)
+ self.assertTrue(is_deepspeed_zero3_enabled())
+
+ # test we can repeat that and with train this time
+ trainer = get_regression_trainer(deepspeed=ds_config_zero3_dict, **kwargs)
+ trainer.train()
+ self.assertTrue(is_deepspeed_zero3_enabled())
+
+ # test zero3 is disabled
+ trainer = get_regression_trainer(deepspeed=ds_config_zero2_dict, **kwargs)
+ self.assertFalse(is_deepspeed_zero3_enabled())
+
+ # check config obj
+ config = deepspeed_config()
+ self.assertTrue(bool(config), "Deepspeed config should be accessible")
+
+ del trainer
+ # now weakref should gc the global and we shouldn't get anything here
+ config = deepspeed_config()
+ self.assertFalse(is_deepspeed_zero3_enabled())
+ self.assertFalse(bool(config), "Deepspeed config should not be accessible")
+
@slow
@require_deepspeed
@require_torch_gpu
class TestDeepSpeedWithLauncher(TestCasePlus):
- """ This class is for testing via an external script - can do multiple gpus """
+ """This class is for testing via an external script - can do multiple gpus"""
# Tests to devise #
#
@@ -557,6 +625,7 @@ def run_trainer(
--adafactor
--source_lang en
--target_lang ro
+ --report_to none
""".split()
args.extend(["--source_prefix", '"translate English to Romanian: "'])
@@ -578,7 +647,7 @@ def run_trainer(
args.extend(
"""
--do_eval
- --max_val_samples 100
+ --max_eval_samples 100
--per_device_eval_batch_size 2
""".split()
)
@@ -620,12 +689,13 @@ def test_clm(self, stage):
--do_train
--do_eval
--max_train_samples 10
- --max_val_samples 10
+ --max_eval_samples 10
--per_device_train_batch_size 5
--per_device_eval_batch_size 5
--num_train_epochs 1
--warmup_steps 8
--block_size 128
+ --report_to none
""".split()
ds_args = f"--deepspeed {self.test_file_dir_str}/ds_config_{stage}.json".split()
diff --git a/tests/extended/test_trainer_ext.py b/tests/extended/test_trainer_ext.py
index 563ea8f059bf..bae358740034 100644
--- a/tests/extended/test_trainer_ext.py
+++ b/tests/extended/test_trainer_ext.py
@@ -191,7 +191,7 @@ def run_trainer(
--output_dir {output_dir}
--overwrite_output_dir
--max_train_samples 8
- --max_val_samples 8
+ --max_eval_samples 8
--max_source_length {max_len}
--max_target_length {max_len}
--val_max_target_length {max_len}
diff --git a/tests/fixtures/ner_pipeline_aligned.json b/tests/fixtures/ner_pipeline_aligned.json
new file mode 100644
index 000000000000..03616893a2e7
--- /dev/null
+++ b/tests/fixtures/ner_pipeline_aligned.json
@@ -0,0 +1,197 @@
+{
+ "first": [
+ {
+ "word": "THe",
+ "score": 0.929590046,
+ "index": 1,
+ "start": 0,
+ "end": 1,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "quasiamerican",
+ "score": 0.62320358,
+ "index": 4,
+ "start": 4,
+ "end": 9,
+ "is_subword": false,
+ "entity": "B-LOC"
+ },
+ {
+ "word": "Mark",
+ "score": 0.9833675,
+ "index": 8,
+ "start": 18,
+ "end": 22,
+ "is_subword": false,
+ "entity": "B-MISC"
+ },
+ {
+ "word": "Musterman",
+ "score": 0.998437107,
+ "index": 9,
+ "start": 23,
+ "end": 27,
+ "is_subword": false,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "works",
+ "score": 0.99952888,
+ "index": 11,
+ "start": 33,
+ "end": 38,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "at",
+ "score": 0.99974287,
+ "index": 12,
+ "start": 39,
+ "end": 41,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "Madeuppity",
+ "score": 0.7194587,
+ "index": 13,
+ "start": 42,
+ "end": 46,
+ "is_subword": false,
+ "entity": "B-ORG"
+ }
+ ],
+ "max": [
+ {
+ "word": "THe",
+ "score": 0.96568632,
+ "index": 1,
+ "start": 0,
+ "end": 1,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "quasiamerican",
+ "score": 0.89459532,
+ "index": 4,
+ "start": 4,
+ "end": 9,
+ "is_subword": false,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "Mark",
+ "score": 0.9833675,
+ "index": 8,
+ "start": 18,
+ "end": 22,
+ "is_subword": false,
+ "entity": "B-MISC"
+ },
+ {
+ "word": "Musterman",
+ "score": 0.998437107,
+ "index": 9,
+ "start": 23,
+ "end": 27,
+ "is_subword": false,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "works",
+ "score": 0.99952888,
+ "index": 11,
+ "start": 33,
+ "end": 38,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "at",
+ "score": 0.99974287,
+ "index": 12,
+ "start": 39,
+ "end": 41,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "Madeuppity",
+ "score": 0.7194587,
+ "index": 13,
+ "start": 42,
+ "end": 46,
+ "is_subword": false,
+ "entity": "B-ORG"
+ }
+ ],
+ "average": [
+ {
+ "word": "THe",
+ "score": 0.821620602,
+ "index": 1,
+ "start": 0,
+ "end": 1,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "quasiamerican",
+ "score": 0.49670231500000006,
+ "index": 4,
+ "start": 4,
+ "end": 9,
+ "is_subword": false,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "Mark",
+ "score": 0.9833675,
+ "index": 8,
+ "start": 18,
+ "end": 22,
+ "is_subword": false,
+ "entity": "B-MISC"
+ },
+ {
+ "word": "Musterman",
+ "score": 0.9982705734999999,
+ "index": 9,
+ "start": 23,
+ "end": 27,
+ "is_subword": false,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "works",
+ "score": 0.99952888,
+ "index": 11,
+ "start": 33,
+ "end": 38,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "at",
+ "score": 0.99974287,
+ "index": 12,
+ "start": 39,
+ "end": 41,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "Madeuppity",
+ "score": 0.6232144095,
+ "index": 13,
+ "start": 42,
+ "end": 46,
+ "is_subword": false,
+ "entity": "B-ORG"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/tests/fixtures/ner_pipeline_aligned_w_subwords.json b/tests/fixtures/ner_pipeline_aligned_w_subwords.json
new file mode 100644
index 000000000000..c8e333391fba
--- /dev/null
+++ b/tests/fixtures/ner_pipeline_aligned_w_subwords.json
@@ -0,0 +1,440 @@
+{
+ "first": [
+ {
+ "word": "T",
+ "score": 0.929590046,
+ "index": 1,
+ "start": 0,
+ "end": 1,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "##H",
+ "score": 0.929590046,
+ "index": 2,
+ "start": 1,
+ "end": 2,
+ "is_subword": true,
+ "entity": "O"
+ },
+ {
+ "word": "##e",
+ "score": 0.929590046,
+ "index": 3,
+ "start": 2,
+ "end": 3,
+ "is_subword": true,
+ "entity": "O"
+ },
+ {
+ "word": "quasi",
+ "score": 0.62320358,
+ "index": 4,
+ "start": 4,
+ "end": 9,
+ "is_subword": false,
+ "entity": "B-LOC"
+ },
+ {
+ "word": "##ame",
+ "score": 0.62320358,
+ "index": 5,
+ "start": 9,
+ "end": 12,
+ "is_subword": true,
+ "entity": "B-LOC"
+ },
+ {
+ "word": "##rica",
+ "score": 0.62320358,
+ "index": 6,
+ "start": 12,
+ "end": 16,
+ "is_subword": true,
+ "entity": "B-LOC"
+ },
+ {
+ "word": "##n",
+ "score": 0.62320358,
+ "index": 7,
+ "start": 16,
+ "end": 17,
+ "is_subword": true,
+ "entity": "B-LOC"
+ },
+ {
+ "word": "Mark",
+ "score": 0.9833675,
+ "index": 8,
+ "start": 18,
+ "end": 22,
+ "is_subword": false,
+ "entity": "B-MISC"
+ },
+ {
+ "word": "Must",
+ "score": 0.998437107,
+ "index": 9,
+ "start": 23,
+ "end": 27,
+ "is_subword": false,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "##erman",
+ "score": 0.998437107,
+ "index": 10,
+ "start": 27,
+ "end": 32,
+ "is_subword": true,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "works",
+ "score": 0.99952888,
+ "index": 11,
+ "start": 33,
+ "end": 38,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "at",
+ "score": 0.99974287,
+ "index": 12,
+ "start": 39,
+ "end": 41,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "Made",
+ "score": 0.7194587,
+ "index": 13,
+ "start": 42,
+ "end": 46,
+ "is_subword": false,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##up",
+ "score": 0.7194587,
+ "index": 14,
+ "start": 46,
+ "end": 48,
+ "is_subword": true,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##pit",
+ "score": 0.7194587,
+ "index": 15,
+ "start": 48,
+ "end": 51,
+ "is_subword": true,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##y",
+ "score": 0.7194587,
+ "index": 16,
+ "start": 51,
+ "end": 52,
+ "is_subword": true,
+ "entity": "B-ORG"
+ }
+ ],
+ "max": [
+ {
+ "word": "T",
+ "score": 0.96568632,
+ "index": 1,
+ "start": 0,
+ "end": 1,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "##H",
+ "score": 0.96568632,
+ "index": 2,
+ "start": 1,
+ "end": 2,
+ "is_subword": true,
+ "entity": "O"
+ },
+ {
+ "word": "##e",
+ "score": 0.96568632,
+ "index": 3,
+ "start": 2,
+ "end": 3,
+ "is_subword": true,
+ "entity": "O"
+ },
+ {
+ "word": "quasi",
+ "score": 0.89459532,
+ "index": 4,
+ "start": 4,
+ "end": 9,
+ "is_subword": false,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "##ame",
+ "score": 0.89459532,
+ "index": 5,
+ "start": 9,
+ "end": 12,
+ "is_subword": true,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "##rica",
+ "score": 0.89459532,
+ "index": 6,
+ "start": 12,
+ "end": 16,
+ "is_subword": true,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "##n",
+ "score": 0.89459532,
+ "index": 7,
+ "start": 16,
+ "end": 17,
+ "is_subword": true,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "Mark",
+ "score": 0.9833675,
+ "index": 8,
+ "start": 18,
+ "end": 22,
+ "is_subword": false,
+ "entity": "B-MISC"
+ },
+ {
+ "word": "Must",
+ "score": 0.998437107,
+ "index": 9,
+ "start": 23,
+ "end": 27,
+ "is_subword": false,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "##erman",
+ "score": 0.998437107,
+ "index": 10,
+ "start": 27,
+ "end": 32,
+ "is_subword": true,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "works",
+ "score": 0.99952888,
+ "index": 11,
+ "start": 33,
+ "end": 38,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "at",
+ "score": 0.99974287,
+ "index": 12,
+ "start": 39,
+ "end": 41,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "Made",
+ "score": 0.7194587,
+ "index": 13,
+ "start": 42,
+ "end": 46,
+ "is_subword": false,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##up",
+ "score": 0.7194587,
+ "index": 14,
+ "start": 46,
+ "end": 48,
+ "is_subword": true,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##pit",
+ "score": 0.7194587,
+ "index": 15,
+ "start": 48,
+ "end": 51,
+ "is_subword": true,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##y",
+ "score": 0.7194587,
+ "index": 16,
+ "start": 51,
+ "end": 52,
+ "is_subword": true,
+ "entity": "B-ORG"
+ }
+ ],
+ "average": [
+ {
+ "word": "T",
+ "score": 0.821620602,
+ "index": 1,
+ "start": 0,
+ "end": 1,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "##H",
+ "score": 0.821620602,
+ "index": 2,
+ "start": 1,
+ "end": 2,
+ "is_subword": true,
+ "entity": "O"
+ },
+ {
+ "word": "##e",
+ "score": 0.821620602,
+ "index": 3,
+ "start": 2,
+ "end": 3,
+ "is_subword": true,
+ "entity": "O"
+ },
+ {
+ "word": "quasi",
+ "score": 0.49670231500000006,
+ "index": 4,
+ "start": 4,
+ "end": 9,
+ "is_subword": false,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "##ame",
+ "score": 0.49670231500000006,
+ "index": 5,
+ "start": 9,
+ "end": 12,
+ "is_subword": true,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "##rica",
+ "score": 0.49670231500000006,
+ "index": 6,
+ "start": 12,
+ "end": 16,
+ "is_subword": true,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "##n",
+ "score": 0.49670231500000006,
+ "index": 7,
+ "start": 16,
+ "end": 17,
+ "is_subword": true,
+ "entity": "I-LOC"
+ },
+ {
+ "word": "Mark",
+ "score": 0.9833675,
+ "index": 8,
+ "start": 18,
+ "end": 22,
+ "is_subword": false,
+ "entity": "B-MISC"
+ },
+ {
+ "word": "Must",
+ "score": 0.9982705734999999,
+ "index": 9,
+ "start": 23,
+ "end": 27,
+ "is_subword": false,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "##erman",
+ "score": 0.9982705734999999,
+ "index": 10,
+ "start": 27,
+ "end": 32,
+ "is_subword": true,
+ "entity": "I-MISC"
+ },
+ {
+ "word": "works",
+ "score": 0.99952888,
+ "index": 11,
+ "start": 33,
+ "end": 38,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "at",
+ "score": 0.99974287,
+ "index": 12,
+ "start": 39,
+ "end": 41,
+ "is_subword": false,
+ "entity": "O"
+ },
+ {
+ "word": "Made",
+ "score": 0.6232144095,
+ "index": 13,
+ "start": 42,
+ "end": 46,
+ "is_subword": false,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##up",
+ "score": 0.6232144095,
+ "index": 14,
+ "start": 46,
+ "end": 48,
+ "is_subword": true,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##pit",
+ "score": 0.6232144095,
+ "index": 15,
+ "start": 48,
+ "end": 51,
+ "is_subword": true,
+ "entity": "B-ORG"
+ },
+ {
+ "word": "##y",
+ "score": 0.6232144095,
+ "index": 16,
+ "start": 51,
+ "end": 52,
+ "is_subword": true,
+ "entity": "B-ORG"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/tests/test_configuration_common.py b/tests/test_configuration_common.py
index eeebf1c9215d..596c73e9891b 100644
--- a/tests/test_configuration_common.py
+++ b/tests/test_configuration_common.py
@@ -99,12 +99,12 @@ def setUpClass(cls):
@classmethod
def tearDownClass(cls):
try:
- cls._api.delete_repo(token=cls._token, name="test-model")
+ cls._api.delete_repo(token=cls._token, name="test-config")
except HTTPError:
pass
try:
- cls._api.delete_repo(token=cls._token, name="test-model-org", organization="valid_org")
+ cls._api.delete_repo(token=cls._token, name="test-config-org", organization="valid_org")
except HTTPError:
pass
@@ -113,9 +113,9 @@ def test_push_to_hub(self):
vocab_size=99, hidden_size=32, num_hidden_layers=5, num_attention_heads=4, intermediate_size=37
)
with tempfile.TemporaryDirectory() as tmp_dir:
- config.save_pretrained(tmp_dir, push_to_hub=True, repo_name="test-model", use_auth_token=self._token)
+ config.save_pretrained(tmp_dir, push_to_hub=True, repo_name="test-config", use_auth_token=self._token)
- new_config = BertConfig.from_pretrained(f"{USER}/test-model")
+ new_config = BertConfig.from_pretrained(f"{USER}/test-config")
for k, v in config.__dict__.items():
if k != "transformers_version":
self.assertEqual(v, getattr(new_config, k))
@@ -129,12 +129,12 @@ def test_push_to_hub_in_organization(self):
config.save_pretrained(
tmp_dir,
push_to_hub=True,
- repo_name="test-model-org",
+ repo_name="test-config-org",
use_auth_token=self._token,
organization="valid_org",
)
- new_config = BertConfig.from_pretrained("valid_org/test-model-org")
+ new_config = BertConfig.from_pretrained("valid_org/test-config-org")
for k, v in config.__dict__.items():
if k != "transformers_version":
self.assertEqual(v, getattr(new_config, k))
diff --git a/tests/test_generation_beam_search.py b/tests/test_generation_beam_search.py
index aa8270c31f2c..fdbe35eafaa4 100644
--- a/tests/test_generation_beam_search.py
+++ b/tests/test_generation_beam_search.py
@@ -59,7 +59,6 @@ def __init__(
def prepare_beam_scorer(self, **kwargs):
return BeamSearchScorer(
batch_size=kwargs.get("batch_size", self.batch_size),
- max_length=kwargs.get("max_length", self.max_length),
num_beams=kwargs.get("num_beams", self.num_beams),
device=torch_device,
length_penalty=kwargs.get("length_penalty", self.length_penalty),
@@ -170,9 +169,7 @@ def cut_expected_tensor(tensor):
def check_beam_scores_finalize(self, input_ids, next_tokens, next_indices, next_scores):
# max_length should be only one more than current input_ids to check that eos is correctly appended
max_length = self.sequence_length + 1
- beam_scorer = self.prepare_beam_scorer(
- num_beam_hyps_to_keep=1, max_length=max_length, length_penalty=1.0, do_early_stopping=False
- )
+ beam_scorer = self.prepare_beam_scorer(num_beam_hyps_to_keep=1, length_penalty=1.0, do_early_stopping=False)
# update beams and append to input_ids
tokens = next_tokens.clone()
@@ -197,6 +194,7 @@ def check_beam_scores_finalize(self, input_ids, next_tokens, next_indices, next_
output_indices,
pad_token_id=self.pad_token_id,
eos_token_id=self.eos_token_id,
+ max_length=max_length,
)
sequences = sequence_output["sequences"]
@@ -225,6 +223,7 @@ def check_beam_scores_finalize(self, input_ids, next_tokens, next_indices, next_
output_indices,
pad_token_id=self.pad_token_id,
eos_token_id=self.eos_token_id,
+ max_length=max_length,
)
sequences = sequence_output["sequences"]
sequence_scores = sequence_output["sequence_scores"]
diff --git a/tests/test_generation_utils.py b/tests/test_generation_utils.py
index 42c44b8c54e8..4a7140d2ca3e 100644
--- a/tests/test_generation_utils.py
+++ b/tests/test_generation_utils.py
@@ -148,7 +148,6 @@ def _get_beam_scorer_and_kwargs(batch_size, max_length, num_return_sequences=1):
}
beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=max_length,
num_beams=beam_kwargs["num_beams"],
device=torch_device,
length_penalty=beam_kwargs["length_penalty"],
@@ -169,7 +168,6 @@ def _get_diverse_beam_scorer_and_kwargs(batch_size, max_length, num_return_seque
}
beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=max_length,
num_beams=beam_kwargs["num_beams"],
device=torch_device,
length_penalty=beam_kwargs["length_penalty"],
@@ -1411,7 +1409,6 @@ def test_max_length_backward_compat_beam_search(self):
beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=max_length,
num_beams=num_beams,
device=torch_device,
)
@@ -1442,7 +1439,6 @@ def test_max_length_backward_compat_group_beam_search(self):
diverse_beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=max_length,
num_beams=num_beams,
device=torch_device,
num_beam_hyps_to_keep=num_return_sequences,
@@ -1502,7 +1498,6 @@ def test_max_length_warning_if_different(self):
# Beam
beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=max_length,
num_beams=num_beams,
device=torch_device,
)
@@ -1520,7 +1515,6 @@ def test_max_length_warning_if_different(self):
# Grouped beam search
diverse_beam_scorer = BeamSearchScorer(
batch_size=batch_size,
- max_length=max_length,
num_beams=num_beams,
device=torch_device,
num_beam_hyps_to_keep=num_return_sequences,
@@ -1535,3 +1529,51 @@ def test_max_length_warning_if_different(self):
max_length=max_length,
**model_kwargs,
)
+
+ def test_beam_search_warning_if_max_length_is_passed(self):
+ article = """Justin Timberlake and Jessica Biel, welcome to parenthood."""
+ bart_tokenizer = BartTokenizer.from_pretrained("sshleifer/bart-tiny-random")
+ bart_model = BartForConditionalGeneration.from_pretrained("sshleifer/bart-tiny-random").to(torch_device)
+
+ batch_size = 1
+ num_beams = 3
+
+ input_ids = bart_tokenizer(article, return_tensors="pt").input_ids.to(torch_device)
+ input_ids = input_ids.expand(num_beams, -1)
+ model_kwargs = bart_model._prepare_encoder_decoder_kwargs_for_generation(input_ids, {})
+
+ stopping_criteria_max_length = 18
+ stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(max_length=stopping_criteria_max_length)])
+
+ with self.assertWarns(UserWarning):
+ beam_scorer = BeamSearchScorer(
+ batch_size=batch_size,
+ num_beams=num_beams,
+ device=torch_device,
+ max_length=10,
+ )
+
+ generated_ids = bart_model.beam_search(
+ input_ids,
+ num_beams=num_beams,
+ stopping_criteria=stopping_criteria,
+ beam_scorer=beam_scorer,
+ **model_kwargs,
+ )
+
+ beam_scorer_no_max_len = BeamSearchScorer(
+ batch_size=batch_size,
+ num_beams=num_beams,
+ device=torch_device,
+ )
+
+ generated_ids_no_max_len = bart_model.beam_search(
+ input_ids,
+ num_beams=num_beams,
+ stopping_criteria=stopping_criteria,
+ beam_scorer=beam_scorer_no_max_len,
+ **model_kwargs,
+ )
+
+ # BeamSearchScorer max_length should not influence "real" max_length
+ self.assertEqual(generated_ids.tolist(), generated_ids_no_max_len.tolist())
diff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py
index b82a8c56641d..d93faa1f6cdf 100755
--- a/tests/test_modeling_common.py
+++ b/tests/test_modeling_common.py
@@ -1122,7 +1122,7 @@ def test_model_parallelization(self):
# a candidate for testing_utils
def get_current_gpu_memory_use():
- """ returns a list of cuda memory allocations per GPU in MBs"""
+ """returns a list of cuda memory allocations per GPU in MBs"""
per_device_memory = []
for id in range(torch.cuda.device_count()):
diff --git a/tests/test_modeling_funnel.py b/tests/test_modeling_funnel.py
index 4435359eb68f..2d59e9f4e410 100644
--- a/tests/test_modeling_funnel.py
+++ b/tests/test_modeling_funnel.py
@@ -42,7 +42,7 @@
class FunnelModelTester:
- """You can also import this e.g, from .test_modeling_funnel import FunnelModelTester """
+ """You can also import this e.g, from .test_modeling_funnel import FunnelModelTester"""
def __init__(
self,
diff --git a/tests/test_modeling_layoutlm.py b/tests/test_modeling_layoutlm.py
index d26bf91cbdfd..a62d13e8fcc6 100644
--- a/tests/test_modeling_layoutlm.py
+++ b/tests/test_modeling_layoutlm.py
@@ -36,7 +36,7 @@
class LayoutLMModelTester:
- """You can also import this e.g from .test_modeling_layoutlm import LayoutLMModelTester """
+ """You can also import this e.g from .test_modeling_layoutlm import LayoutLMModelTester"""
def __init__(
self,
diff --git a/tests/test_modeling_lxmert.py b/tests/test_modeling_lxmert.py
index b03cc31335d9..451db8089a5a 100644
--- a/tests/test_modeling_lxmert.py
+++ b/tests/test_modeling_lxmert.py
@@ -40,7 +40,7 @@
class LxmertModelTester:
- """You can also import this e.g from .test_modeling_bart import BartModelTester """
+ """You can also import this e.g from .test_modeling_bart import BartModelTester"""
def __init__(
self,
diff --git a/tests/test_modeling_tapas.py b/tests/test_modeling_tapas.py
index b36147d5586f..40bdba0e7079 100644
--- a/tests/test_modeling_tapas.py
+++ b/tests/test_modeling_tapas.py
@@ -63,7 +63,7 @@
class TapasModelTester:
- """You can also import this e.g from .test_modeling_tapas import TapasModelTester """
+ """You can also import this e.g from .test_modeling_tapas import TapasModelTester"""
def __init__(
self,
diff --git a/tests/test_modeling_tf_bart.py b/tests/test_modeling_tf_bart.py
index 33aad30be978..e88659b9887d 100644
--- a/tests/test_modeling_tf_bart.py
+++ b/tests/test_modeling_tf_bart.py
@@ -147,6 +147,7 @@ def prepare_bart_inputs_dict(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
):
if attention_mask is None:
attention_mask = tf.cast(tf.math.not_equal(input_ids, config.pad_token_id), tf.int8)
@@ -162,13 +163,16 @@ def prepare_bart_inputs_dict(
head_mask = tf.ones((config.encoder_layers, config.encoder_attention_heads))
if decoder_head_mask is None:
decoder_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
+ if cross_attn_head_mask is None:
+ cross_attn_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
return {
"input_ids": input_ids,
"decoder_input_ids": decoder_input_ids,
"attention_mask": attention_mask,
"decoder_attention_mask": decoder_attention_mask,
"head_mask": head_mask,
- "decoder_head_mask": head_mask,
+ "decoder_head_mask": decoder_head_mask,
+ "cross_attn_head_mask": cross_attn_head_mask,
}
diff --git a/tests/test_modeling_tf_blenderbot.py b/tests/test_modeling_tf_blenderbot.py
index 39e448f277e5..3870f1dff7e6 100644
--- a/tests/test_modeling_tf_blenderbot.py
+++ b/tests/test_modeling_tf_blenderbot.py
@@ -146,6 +146,7 @@ def prepare_blenderbot_inputs_dict(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
):
if attention_mask is None:
attention_mask = tf.cast(tf.math.not_equal(input_ids, config.pad_token_id), tf.int8)
@@ -161,6 +162,8 @@ def prepare_blenderbot_inputs_dict(
head_mask = tf.ones((config.encoder_layers, config.encoder_attention_heads))
if decoder_head_mask is None:
decoder_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
+ if cross_attn_head_mask is None:
+ cross_attn_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
return {
"input_ids": input_ids,
"decoder_input_ids": decoder_input_ids,
@@ -168,6 +171,7 @@ def prepare_blenderbot_inputs_dict(
"decoder_attention_mask": decoder_attention_mask,
"head_mask": head_mask,
"decoder_head_mask": decoder_head_mask,
+ "cross_attn_head_mask": cross_attn_head_mask,
}
diff --git a/tests/test_modeling_tf_blenderbot_small.py b/tests/test_modeling_tf_blenderbot_small.py
index fc49288abfb6..2d99a76ea2c3 100644
--- a/tests/test_modeling_tf_blenderbot_small.py
+++ b/tests/test_modeling_tf_blenderbot_small.py
@@ -146,6 +146,7 @@ def prepare_blenderbot_small_inputs_dict(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
):
if attention_mask is None:
attention_mask = tf.cast(tf.math.not_equal(input_ids, config.pad_token_id), tf.int8)
@@ -161,6 +162,8 @@ def prepare_blenderbot_small_inputs_dict(
head_mask = tf.ones((config.encoder_layers, config.encoder_attention_heads))
if decoder_head_mask is None:
decoder_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
+ if cross_attn_head_mask is None:
+ cross_attn_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
return {
"input_ids": input_ids,
"decoder_input_ids": decoder_input_ids,
@@ -168,6 +171,7 @@ def prepare_blenderbot_small_inputs_dict(
"decoder_attention_mask": decoder_attention_mask,
"head_mask": head_mask,
"decoder_head_mask": decoder_head_mask,
+ "cross_attn_head_mask": cross_attn_head_mask,
}
diff --git a/tests/test_modeling_tf_common.py b/tests/test_modeling_tf_common.py
index 5052a1a0a619..36ce1fbf17c6 100644
--- a/tests/test_modeling_tf_common.py
+++ b/tests/test_modeling_tf_common.py
@@ -190,8 +190,12 @@ def test_forward_signature(self):
"decoder_attention_mask",
]
expected_arg_names.extend(
- ["head_mask", "decoder_head_mask", "encoder_outputs"]
- if "head_mask" and "decoder_head_mask" in arg_names
+ ["head_mask", "decoder_head_mask"] if "head_mask" and "decoder_head_mask" in arg_names else []
+ )
+ # Necessary to handle BART with newly added cross_attn_head_mask
+ expected_arg_names.extend(
+ ["cross_attn_head_mask", "encoder_outputs"]
+ if "cross_attn_head_mask" in arg_names
else ["encoder_outputs"]
)
self.assertListEqual(arg_names[: len(expected_arg_names)], expected_arg_names)
@@ -512,6 +516,8 @@ def test_train_pipeline_custom_model(self):
del inputs_dict["head_mask"]
if "decoder_head_mask" in inputs_dict:
del inputs_dict["decoder_head_mask"]
+ if "cross_attn_head_mask" in inputs_dict:
+ del inputs_dict["cross_attn_head_mask"]
tf_main_layer_classes = set(
module_member
for model_class in self.all_model_classes
@@ -639,7 +645,7 @@ def test_attention_outputs(self):
def check_decoder_attentions_output(outputs):
out_len = len(outputs)
- self.assertEqual(out_len % 2, 0)
+ self.assertEqual(min(out_len % 2, out_len % 5), 0) # differentiation due to newly added cross_attentions
decoder_attentions = outputs.decoder_attentions
self.assertEqual(len(decoder_attentions), self.model_tester.num_hidden_layers)
self.assertListEqual(
@@ -733,6 +739,8 @@ def prepare_layer_head_mask(i, attention_heads, num_hidden_layers):
arg_names = [*signature.parameters.keys()]
if "decoder_head_mask" in arg_names: # necessary diferentiation because of T5 model
inputs["decoder_head_mask"] = head_mask
+ if "cross_attn_head_mask" in arg_names:
+ inputs["cross_attn_head_mask"] = head_mask
outputs = model(**inputs, return_dict=True)
@@ -757,6 +765,8 @@ def check_attentions_validity(attentions):
if model.config.is_encoder_decoder:
check_attentions_validity(outputs.encoder_attentions)
check_attentions_validity(outputs.decoder_attentions)
+ if "cross_attn_head_mask" in arg_names:
+ check_attentions_validity(outputs.cross_attentions)
else:
check_attentions_validity(outputs.attentions)
@@ -1347,12 +1357,12 @@ def setUpClass(cls):
@classmethod
def tearDownClass(cls):
try:
- cls._api.delete_repo(token=cls._token, name="test-model")
+ cls._api.delete_repo(token=cls._token, name="test-model-tf")
except HTTPError:
pass
try:
- cls._api.delete_repo(token=cls._token, name="test-model-org", organization="valid_org")
+ cls._api.delete_repo(token=cls._token, name="test-model-tf-org", organization="valid_org")
except HTTPError:
pass
@@ -1364,9 +1374,9 @@ def test_push_to_hub(self):
# Make sure model is properly initialized
_ = model(model.dummy_inputs)
with tempfile.TemporaryDirectory() as tmp_dir:
- model.save_pretrained(tmp_dir, push_to_hub=True, repo_name="test-model", use_auth_token=self._token)
+ model.save_pretrained(tmp_dir, push_to_hub=True, repo_name="test-model-tf", use_auth_token=self._token)
- new_model = TFBertModel.from_pretrained(f"{USER}/test-model")
+ new_model = TFBertModel.from_pretrained(f"{USER}/test-model-tf")
models_equal = True
for p1, p2 in zip(model.weights, new_model.weights):
if tf.math.reduce_sum(tf.math.abs(p1 - p2)) > 0:
@@ -1382,12 +1392,12 @@ def test_push_to_hub_in_organization(self):
model.save_pretrained(
tmp_dir,
push_to_hub=True,
- repo_name="test-model-org",
+ repo_name="test-model-tf-org",
use_auth_token=self._token,
organization="valid_org",
)
- new_model = TFBertModel.from_pretrained("valid_org/test-model-org")
+ new_model = TFBertModel.from_pretrained("valid_org/test-model-tf-org")
models_equal = True
for p1, p2 in zip(model.weights, new_model.weights):
if tf.math.reduce_sum(tf.math.abs(p1 - p2)) > 0:
diff --git a/tests/test_modeling_tf_funnel.py b/tests/test_modeling_tf_funnel.py
index dc13ed725c0f..93a811f24bdb 100644
--- a/tests/test_modeling_tf_funnel.py
+++ b/tests/test_modeling_tf_funnel.py
@@ -39,7 +39,7 @@
class TFFunnelModelTester:
- """You can also import this e.g, from .test_modeling_funnel import FunnelModelTester """
+ """You can also import this e.g, from .test_modeling_funnel import FunnelModelTester"""
def __init__(
self,
diff --git a/tests/test_modeling_tf_marian.py b/tests/test_modeling_tf_marian.py
index ccea3b79cfb1..3db80bccfe0b 100644
--- a/tests/test_modeling_tf_marian.py
+++ b/tests/test_modeling_tf_marian.py
@@ -148,6 +148,7 @@ def prepare_marian_inputs_dict(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
):
if attention_mask is None:
attention_mask = tf.cast(tf.math.not_equal(input_ids, config.pad_token_id), tf.int8)
@@ -163,6 +164,8 @@ def prepare_marian_inputs_dict(
head_mask = tf.ones((config.encoder_layers, config.encoder_attention_heads))
if decoder_head_mask is None:
decoder_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
+ if cross_attn_head_mask is None:
+ cross_attn_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
return {
"input_ids": input_ids,
"decoder_input_ids": decoder_input_ids,
@@ -170,6 +173,7 @@ def prepare_marian_inputs_dict(
"decoder_attention_mask": decoder_attention_mask,
"head_mask": head_mask,
"decoder_head_mask": decoder_head_mask,
+ "cross_attn_head_mask": cross_attn_head_mask,
}
diff --git a/tests/test_modeling_tf_mbart.py b/tests/test_modeling_tf_mbart.py
index 502be625e78a..e69aee3c0755 100644
--- a/tests/test_modeling_tf_mbart.py
+++ b/tests/test_modeling_tf_mbart.py
@@ -150,6 +150,7 @@ def prepare_mbart_inputs_dict(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
):
if attention_mask is None:
attention_mask = tf.cast(tf.math.not_equal(input_ids, config.pad_token_id), tf.int8)
@@ -165,13 +166,16 @@ def prepare_mbart_inputs_dict(
head_mask = tf.ones((config.encoder_layers, config.encoder_attention_heads))
if decoder_head_mask is None:
decoder_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
+ if cross_attn_head_mask is None:
+ cross_attn_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
return {
"input_ids": input_ids,
"decoder_input_ids": decoder_input_ids,
"attention_mask": attention_mask,
"decoder_attention_mask": decoder_attention_mask,
"head_mask": head_mask,
- "decoder_head_mask": head_mask,
+ "decoder_head_mask": decoder_head_mask,
+ "cross_attn_head_mask": cross_attn_head_mask,
}
diff --git a/tests/test_modeling_tf_pegasus.py b/tests/test_modeling_tf_pegasus.py
index 2be4556425e6..4dc4e9ae9cb7 100644
--- a/tests/test_modeling_tf_pegasus.py
+++ b/tests/test_modeling_tf_pegasus.py
@@ -146,6 +146,7 @@ def prepare_pegasus_inputs_dict(
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
+ cross_attn_head_mask=None,
):
if attention_mask is None:
attention_mask = tf.cast(tf.math.not_equal(input_ids, config.pad_token_id), tf.int8)
@@ -161,6 +162,8 @@ def prepare_pegasus_inputs_dict(
head_mask = tf.ones((config.encoder_layers, config.encoder_attention_heads))
if decoder_head_mask is None:
decoder_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
+ if cross_attn_head_mask is None:
+ cross_attn_head_mask = tf.ones((config.decoder_layers, config.decoder_attention_heads))
return {
"input_ids": input_ids,
"decoder_input_ids": decoder_input_ids,
@@ -168,6 +171,7 @@ def prepare_pegasus_inputs_dict(
"decoder_attention_mask": decoder_attention_mask,
"head_mask": head_mask,
"decoder_head_mask": decoder_head_mask,
+ "cross_attn_head_mask": cross_attn_head_mask,
}
diff --git a/tests/test_pipelines_token_classification.py b/tests/test_pipelines_token_classification.py
index 756ccbf52dd5..925ea8b76efe 100644
--- a/tests/test_pipelines_token_classification.py
+++ b/tests/test_pipelines_token_classification.py
@@ -12,18 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+import json
+import os
import unittest
-from transformers import AutoTokenizer, is_torch_available, pipeline
+import numpy as np
+
+from transformers import AutoTokenizer, pipeline
from transformers.pipelines import Pipeline, TokenClassificationArgumentHandler
from transformers.testing_utils import require_tf, require_torch, slow
from .test_pipelines_common import CustomInputPipelineCommonMixin
-if is_torch_available():
- import numpy as np
-
VALID_INPUTS = ["A simple string", ["list of strings", "A simple string that is quite a bit longer"]]
@@ -190,6 +191,329 @@ def _test_pipeline(self, nlp: Pipeline):
],
]
+ ungrouped_inputs_all_scores = [
+ {
+ "word": "T",
+ "score": np.array(
+ [
+ 9.29590046e-01,
+ 1.67631579e-03,
+ 1.13618866e-04,
+ 5.24316654e-02,
+ 2.45529664e-04,
+ 1.76369259e-03,
+ 7.47492450e-05,
+ 1.37843387e-02,
+ 3.20013991e-04,
+ ]
+ ),
+ "index": 1,
+ "start": 0,
+ "end": 1,
+ "is_subword": False,
+ },
+ {
+ "word": "##H",
+ "score": np.array(
+ [
+ 9.6568632e-01,
+ 1.0918044e-03,
+ 1.3044178e-04,
+ 1.8581267e-02,
+ 3.0361942e-04,
+ 5.9505191e-04,
+ 1.3611461e-04,
+ 1.3115727e-02,
+ 3.5968653e-04,
+ ]
+ ),
+ "index": 2,
+ "start": 1,
+ "end": 2,
+ "is_subword": True,
+ },
+ {
+ "word": "##e",
+ "score": np.array(
+ [
+ 5.6958544e-01,
+ 6.7770253e-03,
+ 3.0190460e-04,
+ 1.6712399e-01,
+ 8.6468906e-04,
+ 3.3661814e-03,
+ 2.7250437e-04,
+ 2.5070670e-01,
+ 1.0015352e-03,
+ ]
+ ),
+ "index": 3,
+ "start": 2,
+ "end": 3,
+ "is_subword": True,
+ },
+ {
+ "word": "quasi",
+ "score": np.array(
+ [
+ 3.6461815e-02,
+ 4.7163994e-04,
+ 7.7644864e-04,
+ 9.0705883e-03,
+ 3.4141801e-03,
+ 5.1569089e-04,
+ 4.5607667e-04,
+ 6.2320358e-01,
+ 3.2563004e-01,
+ ]
+ ),
+ "index": 4,
+ "start": 4,
+ "end": 9,
+ "is_subword": False,
+ },
+ {
+ "word": "##ame",
+ "score": np.array(
+ [
+ 0.15970601,
+ 0.00174455,
+ 0.00181823,
+ 0.00997834,
+ 0.00794088,
+ 0.00160232,
+ 0.00122593,
+ 0.28658998,
+ 0.5293938,
+ ]
+ ),
+ "index": 5,
+ "start": 9,
+ "end": 12,
+ "is_subword": True,
+ },
+ {
+ "word": "##rica",
+ "score": np.array(
+ [
+ 6.4172082e-02,
+ 1.5784599e-03,
+ 1.1795119e-03,
+ 1.3267660e-02,
+ 2.3705978e-03,
+ 1.2073644e-03,
+ 6.1625458e-04,
+ 6.7841798e-01,
+ 2.3719010e-01,
+ ]
+ ),
+ "index": 6,
+ "start": 12,
+ "end": 16,
+ "is_subword": True,
+ },
+ {
+ "word": "##n",
+ "score": np.array(
+ [
+ 2.8728947e-02,
+ 3.4915505e-04,
+ 8.2594587e-04,
+ 3.7162553e-03,
+ 4.2545195e-03,
+ 6.7599339e-04,
+ 5.5553199e-04,
+ 6.6298299e-02,
+ 8.9459532e-01,
+ ]
+ ),
+ "index": 7,
+ "start": 16,
+ "end": 17,
+ "is_subword": True,
+ },
+ {
+ "word": "Mark",
+ "score": np.array(
+ [
+ 3.5328791e-03,
+ 9.8336750e-01,
+ 8.3820109e-04,
+ 7.2647040e-03,
+ 4.9874303e-04,
+ 5.9280143e-04,
+ 2.9107687e-04,
+ 2.3955773e-03,
+ 1.2184654e-03,
+ ]
+ ),
+ "index": 8,
+ "start": 18,
+ "end": 22,
+ "is_subword": False,
+ },
+ {
+ "word": "Must",
+ "score": np.array(
+ [
+ 2.59605236e-04,
+ 2.98619037e-04,
+ 9.98437107e-01,
+ 1.12211506e-04,
+ 1.69078063e-04,
+ 2.54273153e-04,
+ 1.28210217e-04,
+ 6.97476498e-05,
+ 2.71162222e-04,
+ ]
+ ),
+ "index": 9,
+ "start": 23,
+ "end": 27,
+ "is_subword": False,
+ },
+ {
+ "word": "##erman",
+ "score": np.array(
+ [
+ 2.4742461e-04,
+ 5.6770945e-04,
+ 9.9810404e-01,
+ 8.5848493e-05,
+ 3.5093780e-04,
+ 2.3541904e-04,
+ 1.4951559e-04,
+ 5.4812852e-05,
+ 2.0431961e-04,
+ ]
+ ),
+ "index": 10,
+ "start": 27,
+ "end": 32,
+ "is_subword": True,
+ },
+ {
+ "word": "works",
+ "score": np.array(
+ [
+ 9.9952888e-01,
+ 3.4134755e-05,
+ 3.2518183e-05,
+ 2.5094318e-05,
+ 2.4778466e-04,
+ 1.6179767e-05,
+ 3.7871898e-05,
+ 1.8624953e-05,
+ 5.8835212e-05,
+ ]
+ ),
+ "index": 11,
+ "start": 33,
+ "end": 38,
+ "is_subword": False,
+ },
+ {
+ "word": "at",
+ "score": np.array(
+ [
+ 9.9974287e-01,
+ 1.8711693e-05,
+ 6.3416733e-06,
+ 3.5407997e-05,
+ 1.3391797e-04,
+ 2.0818115e-05,
+ 1.8278386e-05,
+ 9.1749180e-06,
+ 1.4530495e-05,
+ ]
+ ),
+ "index": 12,
+ "start": 39,
+ "end": 41,
+ "is_subword": False,
+ },
+ {
+ "word": "Made",
+ "score": np.array(
+ [
+ 1.4745704e-03,
+ 3.5883293e-03,
+ 7.5529744e-05,
+ 2.7471399e-01,
+ 2.6137877e-04,
+ 7.1945870e-01,
+ 5.7339250e-05,
+ 3.3724433e-04,
+ 3.2908196e-05,
+ ]
+ ),
+ "index": 13,
+ "start": 42,
+ "end": 46,
+ "is_subword": False,
+ },
+ {
+ "word": "##up",
+ "score": np.array(
+ [
+ 1.9155587e-03,
+ 5.6133829e-03,
+ 8.0013328e-05,
+ 3.6844590e-01,
+ 3.8169319e-04,
+ 6.2301862e-01,
+ 8.4445433e-05,
+ 4.2535455e-04,
+ 3.5037294e-05,
+ ]
+ ),
+ "index": 14,
+ "start": 46,
+ "end": 48,
+ "is_subword": True,
+ },
+ {
+ "word": "##pit",
+ "score": np.array(
+ [
+ 1.7833502e-03,
+ 3.1480102e-03,
+ 7.2892937e-05,
+ 4.9196857e-01,
+ 2.8379925e-04,
+ 5.0234205e-01,
+ 7.0629183e-05,
+ 2.9715578e-04,
+ 3.3520439e-05,
+ ]
+ ),
+ "index": 15,
+ "start": 48,
+ "end": 51,
+ "is_subword": True,
+ },
+ {
+ "word": "##y",
+ "score": np.array(
+ [
+ 8.12840555e-03,
+ 1.23405308e-02,
+ 3.25592933e-04,
+ 3.28718781e-01,
+ 1.23740966e-03,
+ 6.48038268e-01,
+ 3.58282792e-04,
+ 7.40314135e-04,
+ 1.12461144e-04,
+ ]
+ ),
+ "index": 16,
+ "start": 51,
+ "end": 52,
+ "is_subword": True,
+ },
+ ]
+
expected_grouped_ner_results = [
[
{
@@ -239,6 +563,18 @@ def _test_pipeline(self, nlp: Pipeline):
],
]
+ expected_aligned_results_filepath = os.path.join(
+ os.path.dirname(os.path.abspath(__file__)), "fixtures/ner_pipeline_aligned.json"
+ )
+ with open(expected_aligned_results_filepath) as expected_aligned_results_file:
+ expected_aligned_results = json.load(expected_aligned_results_file)
+
+ expected_aligned_results_w_subword_filepath = os.path.join(
+ os.path.dirname(os.path.abspath(__file__)), "fixtures/ner_pipeline_aligned_w_subwords.json"
+ )
+ with open(expected_aligned_results_w_subword_filepath) as expected_aligned_results_w_subword_file:
+ expected_aligned_results_w_subword = json.load(expected_aligned_results_w_subword_file)
+
self.assertIsNotNone(nlp)
mono_result = nlp(VALID_INPUTS[0])
@@ -271,6 +607,11 @@ def _test_pipeline(self, nlp: Pipeline):
ungrouped_ner_inputs, expected_grouped_ner_results_w_subword
):
self.assertEqual(nlp.group_entities(ungrouped_input), grouped_result)
+ aligned_results = nlp.set_subwords_label(ungrouped_inputs_all_scores)
+ if nlp.ignore_subwords:
+ self.assertEqual(aligned_results, expected_aligned_results[nlp.aggregation_strategy.value])
+ else:
+ self.assertEqual(aligned_results, expected_aligned_results_w_subword[nlp.aggregation_strategy.value])
@require_tf
def test_tf_only(self):
@@ -312,6 +653,35 @@ def test_tf_small_ignore_subwords_available_for_fast_tokenizers(self):
)
self._test_pipeline(nlp)
+ for model_name in self.small_models:
+ tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
+ nlp = pipeline(task="ner", model=model_name, tokenizer=tokenizer, framework="tf")
+ self._test_pipeline(nlp)
+
+ @require_tf
+ def test_tf_small_subwords_aggregation_available_for_fast_tokenizers(self):
+ for model_name in self.small_models:
+ tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
+ for strategy in ["first", "max", "average"]:
+ nlp = pipeline(
+ task="ner",
+ model=model_name,
+ tokenizer=tokenizer,
+ framework="tf",
+ aggregation_strategy=strategy,
+ ignore_subwords=True,
+ )
+ self._test_pipeline(nlp)
+ nlp = pipeline(
+ task="ner",
+ model=model_name,
+ tokenizer=tokenizer,
+ framework="tf",
+ aggregation_strategy=strategy,
+ ignore_subwords=False,
+ )
+ self._test_pipeline(nlp)
+
@require_torch
def test_pt_ignore_subwords_slow_tokenizer_raises(self):
for model_name in self.small_models:
diff --git a/tests/test_tokenization_common.py b/tests/test_tokenization_common.py
index febb9a05c08a..aa6caa35d8f7 100644
--- a/tests/test_tokenization_common.py
+++ b/tests/test_tokenization_common.py
@@ -58,7 +58,7 @@
def filter_non_english(_, pretrained_name: str):
- """ Filter all the model for non-english language """
+ """Filter all the model for non-english language"""
return not any([lang in pretrained_name for lang in NON_ENGLISH_TAGS])
@@ -2874,7 +2874,7 @@ def test_compare_prepare_for_model(self):
@is_staging_test
-class TokenzierPushToHubTester(unittest.TestCase):
+class TokenizerPushToHubTester(unittest.TestCase):
vocab_tokens = ["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]", "bla", "blou"]
@classmethod
@@ -2885,12 +2885,12 @@ def setUpClass(cls):
@classmethod
def tearDownClass(cls):
try:
- cls._api.delete_repo(token=cls._token, name="test-model")
+ cls._api.delete_repo(token=cls._token, name="test-tokenizer")
except HTTPError:
pass
try:
- cls._api.delete_repo(token=cls._token, name="test-model-org", organization="valid_org")
+ cls._api.delete_repo(token=cls._token, name="test-tokenizer-org", organization="valid_org")
except HTTPError:
pass
@@ -2900,9 +2900,11 @@ def test_push_to_hub(self):
with open(vocab_file, "w", encoding="utf-8") as vocab_writer:
vocab_writer.write("".join([x + "\n" for x in self.vocab_tokens]))
tokenizer = BertTokenizer(vocab_file)
- tokenizer.save_pretrained(tmp_dir, push_to_hub=True, repo_name="test-model", use_auth_token=self._token)
+ tokenizer.save_pretrained(
+ tmp_dir, push_to_hub=True, repo_name="test-tokenizer", use_auth_token=self._token
+ )
- new_tokenizer = BertTokenizer.from_pretrained(f"{USER}/test-model")
+ new_tokenizer = BertTokenizer.from_pretrained(f"{USER}/test-tokenizer")
self.assertDictEqual(new_tokenizer.vocab, tokenizer.vocab)
def test_push_to_hub_in_organization(self):
@@ -2914,10 +2916,10 @@ def test_push_to_hub_in_organization(self):
tokenizer.save_pretrained(
tmp_dir,
push_to_hub=True,
- repo_name="test-model-org",
+ repo_name="test-tokenizer-org",
use_auth_token=self._token,
organization="valid_org",
)
- new_tokenizer = BertTokenizer.from_pretrained("valid_org/test-model-org")
+ new_tokenizer = BertTokenizer.from_pretrained("valid_org/test-tokenizer-org")
self.assertDictEqual(new_tokenizer.vocab, tokenizer.vocab)
diff --git a/tests/test_tokenization_fsmt.py b/tests/test_tokenization_fsmt.py
index 8675cc0ffbb2..276941f59462 100644
--- a/tests/test_tokenization_fsmt.py
+++ b/tests/test_tokenization_fsmt.py
@@ -100,7 +100,7 @@ def test_online_tokenizer_config(self):
self.assertEqual(tokenizer.tgt_vocab_size, 21)
def test_full_tokenizer(self):
- """ Adapted from Sennrich et al. 2015 and https://github.com/rsennrich/subword-nmt """
+ """Adapted from Sennrich et al. 2015 and https://github.com/rsennrich/subword-nmt"""
tokenizer = FSMTTokenizer(self.langs, self.src_vocab_file, self.tgt_vocab_file, self.merges_file)
text = "lower"
diff --git a/tests/test_tokenization_layoutlm.py b/tests/test_tokenization_layoutlm.py
index 89aac1355c20..79831cd30c4d 100644
--- a/tests/test_tokenization_layoutlm.py
+++ b/tests/test_tokenization_layoutlm.py
@@ -70,5 +70,5 @@ def test_full_tokenizer(self):
self.assertListEqual(tokenizer.convert_tokens_to_ids(tokens), [7, 4, 5, 10, 8, 9])
def test_special_tokens_as_you_expect(self):
- """If you are training a seq2seq model that expects a decoder_prefix token make sure it is prepended to decoder_input_ids """
+ """If you are training a seq2seq model that expects a decoder_prefix token make sure it is prepended to decoder_input_ids"""
pass
diff --git a/tests/test_tokenization_xlm.py b/tests/test_tokenization_xlm.py
index e39426e850b8..cf0296ddd9b0 100644
--- a/tests/test_tokenization_xlm.py
+++ b/tests/test_tokenization_xlm.py
@@ -72,7 +72,7 @@ def get_input_output_texts(self, tokenizer):
return input_text, output_text
def test_full_tokenizer(self):
- """ Adapted from Sennrich et al. 2015 and https://github.com/rsennrich/subword-nmt """
+ """Adapted from Sennrich et al. 2015 and https://github.com/rsennrich/subword-nmt"""
tokenizer = XLMTokenizer(self.vocab_file, self.merges_file)
text = "lower"
diff --git a/tests/test_trainer.py b/tests/test_trainer.py
index 2edce418a036..68a15ae67350 100644
--- a/tests/test_trainer.py
+++ b/tests/test_trainer.py
@@ -213,16 +213,21 @@ def get_regression_trainer(a=0, b=0, double_output=False, train_len=64, eval_len
label_names = kwargs.get("label_names", None)
train_dataset = RegressionDataset(length=train_len, label_names=label_names)
eval_dataset = RegressionDataset(length=eval_len, label_names=label_names)
- if pretrained:
- config = RegressionModelConfig(a=a, b=b, double_output=double_output)
- model = RegressionPreTrainedModel(config)
+
+ model_init = kwargs.pop("model_init", None)
+ if model_init is not None:
+ model = None
else:
- model = RegressionModel(a=a, b=b, double_output=double_output)
+ if pretrained:
+ config = RegressionModelConfig(a=a, b=b, double_output=double_output)
+ model = RegressionPreTrainedModel(config)
+ else:
+ model = RegressionModel(a=a, b=b, double_output=double_output)
+
compute_metrics = kwargs.pop("compute_metrics", None)
data_collator = kwargs.pop("data_collator", None)
optimizers = kwargs.pop("optimizers", (None, None))
output_dir = kwargs.pop("output_dir", "./regression")
- model_init = kwargs.pop("model_init", None)
args = RegressionTrainingArguments(output_dir, a=a, b=b, **kwargs)
return Trainer(
@@ -1099,12 +1104,12 @@ def setUpClass(cls):
@classmethod
def tearDownClass(cls):
try:
- cls._api.delete_repo(token=cls._token, name="test-model")
+ cls._api.delete_repo(token=cls._token, name="test-trainer")
except HTTPError:
pass
try:
- cls._api.delete_repo(token=cls._token, name="test-model-org", organization="valid_org")
+ cls._api.delete_repo(token=cls._token, name="test-trainer-org", organization="valid_org")
except HTTPError:
pass
@@ -1112,14 +1117,14 @@ def test_push_to_hub(self):
with tempfile.TemporaryDirectory() as tmp_dir:
trainer = get_regression_trainer(output_dir=tmp_dir)
trainer.save_model()
- url = trainer.push_to_hub(repo_name="test-model", use_auth_token=self._token)
+ url = trainer.push_to_hub(repo_name="test-trainer", use_auth_token=self._token)
# Extract repo_name from the url
re_search = re.search(ENDPOINT_STAGING + r"/([^/]+/[^/]+)/", url)
self.assertTrue(re_search is not None)
repo_name = re_search.groups()[0]
- self.assertEqual(repo_name, f"{USER}/test-model")
+ self.assertEqual(repo_name, f"{USER}/test-trainer")
model = RegressionPreTrainedModel.from_pretrained(repo_name)
self.assertEqual(model.a.item(), trainer.model.a.item())
@@ -1129,15 +1134,17 @@ def test_push_to_hub_in_organization(self):
with tempfile.TemporaryDirectory() as tmp_dir:
trainer = get_regression_trainer(output_dir=tmp_dir)
trainer.save_model()
- url = trainer.push_to_hub(repo_name="test-model-org", organization="valid_org", use_auth_token=self._token)
+ url = trainer.push_to_hub(
+ repo_name="test-trainer-org", organization="valid_org", use_auth_token=self._token
+ )
# Extract repo_name from the url
re_search = re.search(ENDPOINT_STAGING + r"/([^/]+/[^/]+)/", url)
self.assertTrue(re_search is not None)
repo_name = re_search.groups()[0]
- self.assertEqual(repo_name, "valid_org/test-model-org")
+ self.assertEqual(repo_name, "valid_org/test-trainer-org")
- model = RegressionPreTrainedModel.from_pretrained("valid_org/test-model-org")
+ model = RegressionPreTrainedModel.from_pretrained("valid_org/test-trainer-org")
self.assertEqual(model.a.item(), trainer.model.a.item())
self.assertEqual(model.b.item(), trainer.model.b.item())
diff --git a/utils/check_copies.py b/utils/check_copies.py
index 3d6ef7adbddf..db1999d22447 100644
--- a/utils/check_copies.py
+++ b/utils/check_copies.py
@@ -33,7 +33,7 @@ def _should_continue(line, indent):
def find_code_in_transformers(object_name):
- """ Find and return the code source code of `object_name`."""
+ """Find and return the code source code of `object_name`."""
parts = object_name.split(".")
i = 0
@@ -193,7 +193,7 @@ def check_copies(overwrite: bool = False):
def get_model_list():
- """ Extracts the model list from the README. """
+ """Extracts the model list from the README."""
# If the introduction or the conclusion of the list change, the prompts may need to be updated.
_start_prompt = "🤗 Transformers currently provides the following architectures"
_end_prompt = "1. Want to contribute a new model?"
@@ -224,7 +224,7 @@ def get_model_list():
def split_long_line_with_indent(line, max_per_line, indent):
- """ Split the `line` so that it doesn't go over `max_per_line` and adds `indent` to new lines. """
+ """Split the `line` so that it doesn't go over `max_per_line` and adds `indent` to new lines."""
words = line.split(" ")
lines = []
current_line = words[0]
@@ -239,7 +239,7 @@ def split_long_line_with_indent(line, max_per_line, indent):
def convert_to_rst(model_list, max_per_line=None):
- """ Convert `model_list` to rst format. """
+ """Convert `model_list` to rst format."""
# Convert **[description](link)** to `description `__
def _rep_link(match):
title, link = match.groups()
@@ -298,7 +298,7 @@ def _find_text_in_file(filename, start_prompt, end_prompt):
def check_model_list_copy(overwrite=False, max_per_line=119):
- """ Check the model lists in the README and index.rst are consistent and maybe `overwrite`. """
+ """Check the model lists in the README and index.rst are consistent and maybe `overwrite`."""
rst_list, start_index, end_index, lines = _find_text_in_file(
filename=os.path.join(PATH_TO_DOCS, "index.rst"),
start_prompt=" This list is updated automatically from the README",
diff --git a/utils/check_dummies.py b/utils/check_dummies.py
index 89965f978421..fb71ea1536cd 100644
--- a/utils/check_dummies.py
+++ b/utils/check_dummies.py
@@ -65,7 +65,7 @@ def find_backend(line):
def read_init():
- """ Read the init and extracts PyTorch, TensorFlow, SentencePiece and Tokenizers objects. """
+ """Read the init and extracts PyTorch, TensorFlow, SentencePiece and Tokenizers objects."""
with open(os.path.join(PATH_TO_TRANSFORMERS, "__init__.py"), "r", encoding="utf-8", newline="\n") as f:
lines = f.readlines()
@@ -101,7 +101,7 @@ def read_init():
def create_dummy_object(name, backend_name):
- """ Create the code for the dummy object corresponding to `name`."""
+ """Create the code for the dummy object corresponding to `name`."""
_pretrained = [
"Config" "ForCausalLM",
"ForConditionalGeneration",
@@ -130,7 +130,7 @@ def create_dummy_object(name, backend_name):
def create_dummy_files():
- """ Create the content of the dummy files. """
+ """Create the content of the dummy files."""
backend_specific_objects = read_init()
# For special correspondence backend to module name as used in the function requires_modulename
dummy_files = {}
@@ -146,7 +146,7 @@ def create_dummy_files():
def check_dummies(overwrite=False):
- """ Check if the dummy files are up to date and maybe `overwrite` with the right content. """
+ """Check if the dummy files are up to date and maybe `overwrite` with the right content."""
dummy_files = create_dummy_files()
# For special correspondence backend to shortcut as used in utils/dummy_xxx_objects.py
short_names = {"torch": "pt"}
diff --git a/utils/check_repo.py b/utils/check_repo.py
index 6f5fd8faf3c0..bd6c9af45b23 100644
--- a/utils/check_repo.py
+++ b/utils/check_repo.py
@@ -119,7 +119,7 @@
# If some modeling modules should be ignored for all checks, they should be added in the nested list
# _ignore_modules of this function.
def get_model_modules():
- """ Get the model modules inside the transformers library. """
+ """Get the model modules inside the transformers library."""
_ignore_modules = [
"modeling_auto",
"modeling_encoder_decoder",
@@ -151,7 +151,7 @@ def get_model_modules():
def get_models(module):
- """ Get the objects in module that are models."""
+ """Get the objects in module that are models."""
models = []
model_classes = (transformers.PreTrainedModel, transformers.TFPreTrainedModel, transformers.FlaxPreTrainedModel)
for attr_name in dir(module):
@@ -166,7 +166,7 @@ def get_models(module):
# If some test_modeling files should be ignored when checking models are all tested, they should be added in the
# nested list _ignore_files of this function.
def get_model_test_files():
- """ Get the model test files."""
+ """Get the model test files."""
_ignore_files = [
"test_modeling_common",
"test_modeling_encoder_decoder",
@@ -187,7 +187,7 @@ def get_model_test_files():
# This is a bit hacky but I didn't find a way to import the test_file as a module and read inside the tester class
# for the all_model_classes variable.
def find_tested_models(test_file):
- """ Parse the content of test_file to detect what's in all_model_classes"""
+ """Parse the content of test_file to detect what's in all_model_classes"""
# This is a bit hacky but I didn't find a way to import the test_file as a module and read inside the class
with open(os.path.join(PATH_TO_TESTS, test_file), "r", encoding="utf-8", newline="\n") as f:
content = f.read()
@@ -205,7 +205,7 @@ def find_tested_models(test_file):
def check_models_are_tested(module, test_file):
- """ Check models defined in module are tested in test_file."""
+ """Check models defined in module are tested in test_file."""
defined_models = get_models(module)
tested_models = find_tested_models(test_file)
if tested_models is None:
@@ -229,7 +229,7 @@ def check_models_are_tested(module, test_file):
def check_all_models_are_tested():
- """ Check all models are properly tested."""
+ """Check all models are properly tested."""
modules = get_model_modules()
test_files = get_model_test_files()
failures = []
@@ -245,7 +245,7 @@ def check_all_models_are_tested():
def get_all_auto_configured_models():
- """ Return the list of all models in at least one auto class."""
+ """Return the list of all models in at least one auto class."""
result = set() # To avoid duplicates we concatenate all model classes in a set.
for attr_name in dir(transformers.models.auto.modeling_auto):
if attr_name.startswith("MODEL_") and attr_name.endswith("MAPPING"):
@@ -271,7 +271,7 @@ def ignore_unautoclassed(model_name):
def check_models_are_auto_configured(module, all_auto_models):
- """ Check models defined in module are each in an auto class."""
+ """Check models defined in module are each in an auto class."""
defined_models = get_models(module)
failures = []
for model_name, _ in defined_models:
@@ -285,7 +285,7 @@ def check_models_are_auto_configured(module, all_auto_models):
def check_all_models_are_auto_configured():
- """ Check all models are each in an auto class."""
+ """Check all models are each in an auto class."""
modules = get_model_modules()
all_auto_models = get_all_auto_configured_models()
failures = []
@@ -301,7 +301,7 @@ def check_all_models_are_auto_configured():
def check_decorator_order(filename):
- """ Check that in the test file `filename` the slow decorator is always last."""
+ """Check that in the test file `filename` the slow decorator is always last."""
with open(filename, "r", encoding="utf-8", newline="\n") as f:
lines = f.readlines()
decorator_before = None
@@ -319,7 +319,7 @@ def check_decorator_order(filename):
def check_all_decorator_order():
- """ Check that in all test files, the slow decorator is always last."""
+ """Check that in all test files, the slow decorator is always last."""
errors = []
for fname in os.listdir(PATH_TO_TESTS):
if fname.endswith(".py"):
@@ -334,7 +334,7 @@ def check_all_decorator_order():
def find_all_documented_objects():
- """ Parse the content of all doc files to detect which classes and functions it documents"""
+ """Parse the content of all doc files to detect which classes and functions it documents"""
documented_obj = []
for doc_file in Path(PATH_TO_DOC).glob("**/*.rst"):
with open(doc_file, "r", encoding="utf-8", newline="\n") as f:
@@ -454,7 +454,7 @@ def ignore_undocumented(name):
def check_all_objects_are_documented():
- """ Check all models are properly documented."""
+ """Check all models are properly documented."""
documented_objs = find_all_documented_objects()
modules = transformers._modules
objects = [c for c in dir(transformers) if c not in modules and not c.startswith("_")]
@@ -467,7 +467,7 @@ def check_all_objects_are_documented():
def check_repo_quality():
- """ Check all models are properly tested and documented."""
+ """Check all models are properly tested and documented."""
print("Checking all models are properly tested.")
check_all_decorator_order()
check_all_models_are_tested()
diff --git a/utils/check_table.py b/utils/check_table.py
index b45daf46edd0..9151040fc938 100644
--- a/utils/check_table.py
+++ b/utils/check_table.py
@@ -159,7 +159,7 @@ def get_model_table_from_auto_modules():
def check_model_table(overwrite=False):
- """ Check the model table in the index.rst is consistent with the state of the lib and maybe `overwrite`. """
+ """Check the model table in the index.rst is consistent with the state of the lib and maybe `overwrite`."""
current_table, start_index, end_index, lines = _find_text_in_file(
filename=os.path.join(PATH_TO_DOCS, "index.rst"),
start_prompt=" This table is updated automatically from the auto module",
diff --git a/utils/style_doc.py b/utils/style_doc.py
index 4da470991243..82341a07c410 100644
--- a/utils/style_doc.py
+++ b/utils/style_doc.py
@@ -431,7 +431,7 @@ def _add_new_lines_before_doc_special_words(text):
def style_rst_file(doc_file, max_len=119, check_only=False):
- """ Style one rst file `doc_file` to `max_len`."""
+ """Style one rst file `doc_file` to `max_len`."""
with open(doc_file, "r", encoding="utf-8", newline="\n") as f:
doc = f.read()