Add Information Gain Filtration algorithm

[mraunak]

What does this PR do?

Adding a new feature for fine-tuning transformer models called Information Gain Filtration 'IGF'

Motivation

The quality of a fine-tuned model depends on the quality of the data samples used for the first few batches. As the process is stochastic, a random seed would influence the quality of the final fine-tuned model. We are proposing a novel and robust fine-tuning method “Information Gain Filtration” (IGF), which filters informative training samples before a fine-tuning (training) and improves the overall training efficiency and final performance of the language model fine-tuning step

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. This can be of interest to @sgugger,

Models:

• gpt2

Examples:

• research_projects/information-gain-filtration: @TuKo,@mraunak

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[LysandreJik]

LGTM, thank you @mraunak!

[LysandreJik]

Could you just run the code quality tool to ensure that the code quality passes? You can install them with the following, from the root of your clone:

pip install -e ".[quality]"

And then run them with:

make fixup

[mraunak]

Running the command: make fixup, gives an error that does not include terms from my PR,

The output with error is shown below. Please guide me on it. Thanks

(igfprnew) mraunak@bcl-main1:~/transformers$ make fixup
No library .py files were modified
python utils/custom_init_isort.py
python utils/style_doc.py src/transformers docs/source --max_len 119
running deps_table_update
updating src/transformers/dependency_versions_table.py
python utils/check_copies.py
python utils/check_table.py
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
python utils/check_dummies.py
python utils/check_repo.py
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Checking all models are included.
Checking all models are public.
Checking all models are properly tested.
Checking all objects are properly documented.
Checking all models are in at least one auto class.
utils/check_repo.py:456: UserWarning: Full quality checks require all backends to be installed (with pip install -e .[dev] in the Transformers repo, the following are missing: PyTorch, TensorFlow, Flax. While it's probably fine as long as you didn't make any change in one of those backends modeling files, you should probably execute the command above to be on the safe side.
warnings.warn(
python utils/check_inits.py
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Traceback (most recent call last):
File "utils/check_inits.py", line 265, in
check_submodules()
File "utils/check_inits.py", line 256, in check_submodules
raise ValueError(
ValueError: The following submodules are not properly registed in the main init of Transformers:

• sagemaker
• activations
• activations_tf
• convert_slow_tokenizer
• deepspeed
• generation_beam_constraints
• generation_beam_search
• generation_flax_logits_process
• generation_flax_utils
• generation_logits_process
• generation_stopping_criteria
• generation_tf_logits_process
• generation_tf_utils
• generation_utils
• image_utils
• keras_callbacks
• modeling_flax_outputs
• modeling_flax_utils
• modeling_outputs
• modeling_tf_outputs
• modeling_tf_utils
• modeling_utils
• optimization
• optimization_tf
• pytorch_utils
• tf_utils
• trainer
• trainer_pt_utils
• trainer_seq2seq
• trainer_tf
• data.datasets
  Make sure they appear somewhere in the keys of _import_structure with an empty list as value.
  make: *** [repo-consistency] Error 1

[TuKo]

@LysandreJik thanks for the suggestion! We were able to correct the quality check issues.
Let us know if you need us to run/test anything else. Thank you!

[LysandreJik]

Thank you for your contributions!

[TuKo]

Thank you for accepting our work!