Natural Instructions

Natural-Instructions is a dataset of various NLP tasks and their language instructions. We have built this data using existing NLP datasets and the instructions that were used to crowdsource them.

• Update (July 2021): Help us expand the instructions!

Dataset

You can download the data on this website: https://instructions.apps.allenai.org/

Model predictions

We have the model predictions for the following models:

predictions/gpt3_outputs

We will add the BART predictions at a later time. The BART predictions, in particular, correspond to a model that was trained on a random subset of tasks and evaluated on the remaining ones.

Evaluation script

The script that we used in our evaluation is included in src/evaluation.py.

How to Evaluate

It requires dataset file path and the prediction file path E.g.

python3 evaluation.py --predictions ../predictions/gpt3_outputs/subtask002_quoref_answer_generation@_Definition_Prompt@0_100.json --dataset ../Dataset_Jsons/subtask002_quoref_answer_generation.json

Filenames in predictions/gpt3_outputs are of the format [taskname]'@'[instruction encoding]'@'[number of examples]'_'[number of instances].json

Encoding the instructions

The encoding function is provided to generate encoded instruction inputs. E.g.

encodeinstruction('subtask003_mctaco_question_generation_event_duration', instruction_structure =['Definition','Prompt'])

Baselines

We have two baselines used in this work:

• GPT-3: we have included the predictions made by our GPT-3 baselines in gpt3_output. If you want to try GPT-3 yourself, you can ask for API access in this link.

• BART: To reproduce our BART predictions, use our encoding function and train a BART model on them

Expanding the data

We're expanding this dataset. Help us with the expansion! See the details here.

How to cite

Feel free to cite us:

@article{mishra2021natural,
  title={Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions},
  author={Mishra, Swaroop and Khashabi, Daniel and Baral, Chitta and Hajishirzi, Hannaneh},
  journal={arXiv preprint arXiv:2104.08773},
  year={2021}
}