Skip to content

[Examples] Added support for test-file in QA examples with no trainer #11510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sgugger merged 8 commits into from
Apr 30, 2021
Merged

[Examples] Added support for test-file in QA examples with no trainer #11510

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6304af4
added support for test-file
bhadreshpsavani Apr 29, 2021
11a487d
fixed typo
bhadreshpsavani Apr 29, 2021
30097e7
added suggested changes
bhadreshpsavani Apr 30, 2021
36745f1
reformatted code
bhadreshpsavani Apr 30, 2021
599a87f
modifed files
bhadreshpsavani Apr 30, 2021
0f2f74a
fix post processing error
bhadreshpsavani Apr 30, 2021
1b0af5f
Trigger CI
bhadreshpsavani Apr 30, 2021
a27bd1a
removed extra lines
bhadreshpsavani Apr 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions examples/pytorch/question-answering/run_qa_beam_search_no_trainer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,9 @@ def parse_args():
parser.add_argument(
"--validation_file", type=str, default=None, help="A csv or a json file containing the validation data."
)
parser.add_argument(
"--test_file", type=str, default=None, help="A csv or a json file containing the Prediction data."
)
parser.add_argument(
"--max_seq_length",
type=int,
Expand Down Expand Up @@ -202,15 +205,23 @@ def parse_args():
args = parser.parse_args()

# Sanity checks
if args.dataset_name is None and args.train_file is None and args.validation_file is None:
raise ValueError("Need either a dataset name or a training/validation file.")
if (
args.dataset_name is None
and args.train_file is None
and args.validation_file is None
and args.test_file is None
):
raise ValueError("Need either a dataset name or a training/validation/test file.")
else:
if args.train_file is not None:
extension = args.train_file.split(".")[-1]
assert extension in ["csv", "json"], "`train_file` should be a csv or a json file."
if args.validation_file is not None:
extension = args.validation_file.split(".")[-1]
assert extension in ["csv", "json"], "`validation_file` should be a csv or a json file."
if args.test_file is not None:
extension = args.test_file.split(".")[-1]
assert extension in ["csv", "json"], "`test_file` should be a csv or a json file."

if args.output_dir is not None:
os.makedirs(args.output_dir, exist_ok=True)
Expand Down Expand Up @@ -263,8 +274,10 @@ def main():
data_files["train"] = args.train_file
if args.validation_file is not None:
data_files["validation"] = args.validation_file
if args.test_file is not None:
data_files["test"] = args.test_file
extension = args.train_file.split(".")[-1]
raw_datasets = load_dataset(extension, data_files=data_files)
raw_datasets = load_dataset(extension, data_files=data_files, field="data")
# See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
# https://huggingface.co/docs/datasets/loading_datasets.html.

Expand Down Expand Up @@ -535,11 +548,11 @@ def prepare_validation_features(examples):
train_dataset, shuffle=True, collate_fn=data_collator, batch_size=args.per_device_train_batch_size
)

eval_dataset.set_format(type="torch", columns=["attention_mask", "input_ids", "token_type_ids"])
eval_dataset.set_format(type="torch", columns=["attention_mask", "input_ids"])

@sgugger sgugger Apr 29, 2021

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the token type IDs should definitely be there, there are the ones that tell the model which part of the input is inside the question and which is inside the context. There are also more columns to pass in this case (the XlnetModelForQuestionAnswering expects more inputs that are prepared in this script: is_impossible, cls_index and p_mask. They all should be listed here. Plus the labels start_logits and end_logits.

The goal of this line is to remove the columns named "example_id" which is not accepted by the model, but the contributor forgot to list all relevant inputs. There is an easier way now that datasets has more functionality, I think you should do:

eval_dataset_for_model = eval_dataset.remove_columns(["example_id"])

and use that when building your DataLoader. Then you can remove the line way below that does

eval_dataset.set_format(type=None, columns=list(eval_dataset.features.keys()))

since eval_dataset did not change.

eval_dataloader = DataLoader(eval_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size)

if args.do_predict:
predict_dataset.set_format(type="torch", columns=["attention_mask", "input_ids", "token_type_ids"])
predict_dataset.set_format(type="torch", columns=["attention_mask", "input_ids"])

@sgugger sgugger Apr 29, 2021

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above here and you can remove the line below

predict_dataset.set_format(type=None, columns=list(predict_dataset.features.keys()))

predict_dataloader = DataLoader(
predict_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size
)
Expand Down
25 changes: 19 additions & 6 deletions examples/pytorch/question-answering/run_qa_no_trainer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,13 @@ def parse_args():
parser.add_argument(
"--preprocessing_num_workers", type=int, default=4, help="A csv or a json file containing the training data."
)
parser.add_argument("--do_predict", action="store_true", help="Eval the question answering model")
parser.add_argument("--do_predict", action="store_true", help="To do prediction on the question answering model")
parser.add_argument(
"--validation_file", type=str, default=None, help="A csv or a json file containing the validation data."
)
parser.add_argument(
"--test_file", type=str, default=None, help="A csv or a json file containing the Prediction data."
)
parser.add_argument(
"--max_seq_length",
type=int,
Expand Down Expand Up @@ -231,15 +234,23 @@ def parse_args():
args = parser.parse_args()

# Sanity checks
if args.dataset_name is None and args.train_file is None and args.validation_file is None:
raise ValueError("Need either a dataset name or a training/validation file.")
if (
args.dataset_name is None
and args.train_file is None
and args.validation_file is None
and args.test_file is None
):
raise ValueError("Need either a dataset name or a training/validation/test file.")
else:
if args.train_file is not None:
extension = args.train_file.split(".")[-1]
assert extension in ["csv", "json"], "`train_file` should be a csv or a json file."
if args.validation_file is not None:
extension = args.validation_file.split(".")[-1]
assert extension in ["csv", "json"], "`validation_file` should be a csv or a json file."
if args.test_file is not None:
extension = args.test_file.split(".")[-1]
assert extension in ["csv", "json"], "`test_file` should be a csv or a json file."

if args.output_dir is not None:
os.makedirs(args.output_dir, exist_ok=True)
Expand Down Expand Up @@ -292,8 +303,10 @@ def main():
data_files["train"] = args.train_file
if args.validation_file is not None:
data_files["validation"] = args.validation_file
if args.test_file is not None:
data_files["test"] = args.test_file
extension = args.train_file.split(".")[-1]
raw_datasets = load_dataset(extension, data_files=data_files)
raw_datasets = load_dataset(extension, data_files=data_files, field="data")
Comment thread
stas00 marked this conversation as resolved.
# See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
# https://huggingface.co/docs/datasets/loading_datasets.html.

Expand Down Expand Up @@ -540,11 +553,11 @@ def prepare_validation_features(examples):
train_dataset, shuffle=True, collate_fn=data_collator, batch_size=args.per_device_train_batch_size
)

eval_dataset.set_format(type="torch", columns=["attention_mask", "input_ids", "token_type_ids"])
eval_dataset.set_format(type="torch", columns=["attention_mask", "input_ids"])

@sgugger sgugger Apr 29, 2021

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as in the other script here.

eval_dataloader = DataLoader(eval_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size)

if args.do_predict:
predict_dataset.set_format(type="torch", columns=["attention_mask", "input_ids", "token_type_ids"])
predict_dataset.set_format(type="torch", columns=["attention_mask", "input_ids"])

@sgugger sgugger Apr 29, 2021

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here too

predict_dataloader = DataLoader(
predict_dataset, collate_fn=data_collator, batch_size=args.per_device_eval_batch_size
)
Expand Down