Skip to content

Latest commit

 

History

History
86 lines (75 loc) · 4.25 KB

readme.md

File metadata and controls

86 lines (75 loc) · 4.25 KB
Raw

PASTA:A Dataset for Modeling Participant States in Narratives

  • Dataset The train, validation and test dataset are in data/tr_data.jsonl, data/val_data.jsonl and data/te_data.jsonl respectively. The fields in the data are:
    • AssignmentId - unique ID of a PASTA response
    • Input.Title - title of the story which is provided to the annotator for generating a PASTA response.
    • Input.storyid - unique (ROC story) ID of the input story.
    • Input.line1 to Input.line5 are the 5 stentences of the input story.
    • Answer.assertion - participant state inferred from the input story
    • Answer.line1.on to Answer.line5.on indicates which are the minimal set of story sentences based on which the annotator inferred the participant state.
    • Answer.mod_assertion - perturbed participant state that is counterfactual to the input story.
    • Answer.mod_line1 to Answer.mod_line5 are the sentences of the modified story that are consistent with the perturbed state.

- **Tasks on ```PASTA```**

  • Story-State Inference

    Task 8

    • Throughout the codebase, the task_id for this task is 8.

    • Finetuning BERT/RoBERTa (base/large)

    CUDA_VISIBLE_DEVICES=0 python ./code/modelling_BERT_RoBERTa.py \
-task_list "8" \                  # taks id of this task, i.e. "8"
-num_epochs 6 \                   # num of training epochs
-eval_steps 1000 \                # training step after which val-set is evaluated
-batch_size 16 \                  # training batch size
-model_type "bert-base-uncased" \ # model type (bert-base/roberta-large etc.)
-random_seed 1231 \               # random seed
-lr 5e-6 \                        # Learning rate of AdamW optim
-task8_setting "1" \              # "1" (default): make inference based on highlighted part of the story, "0" make inference based on entire story 
-wt_decay 1e-6 \                  # weight decay of the AdamW optim
> ./logs/bert-base-task8.log &
    • Finetuning T5 (base/large)
    python ./code/modelling_t5.py \
-task_list "8" \            
-num_epochs 7 \             # num of training epochs
-eval_steps 1000 \          # training step after which val-set is evaluated
-batch_size 16 \            # training batch size
-model_type "t5-base" \     # model type (t5-base/t5-large etc.)
-random_seed 0 \            # random seed
-lr 1e-4 \                  # Learning rate of AdamW optim
-wt_decay 1e-6              # weight decay of the AdamW optim

  • Story Revision from Counterfactual/State Change Generation


  • Task 6


  • Task 7

    • Finetuning T5 (base/large)
    CUDA_VISIBLE_DEVICES=1 nohup python ./code/modelling_t5.py \
-task_list "7" \      # "6" (story revision task) or "7" (state change task)
-num_epochs 2 \
-eval_steps 1000 \
-batch_size 12 \
-model_type "t5-base" \
-random_seed 0 \
-lr 1e-4 \
-wt_decay 1e-6 > ./logs/t5-base-task7.log &
    • GPT3 few-shot prompting code can be found in ./code/gpt3_code.ipynb

  • Inference on validation/test set from a checkpoint
    Code generates model prediction, saves it, and generate the automatic metric based on ground truth & model prediction.(acc/contrastive acc for classificaion, BERTscore for generation task)

    CUDA_VISIBLE_DEVICES=0 nohup python ./code/inference_T5.py \
-task_list "6" \
-model_type "t5-base" \
-checkpoint "model_checkpoint_path (.pt)" \
-task8_setting "1" > logs/_t5b_t_6_op.log &

CUDA_VISIBLE_DEVICES=0 nohup python ./code/inference_BERT.py \
-task_list "8" \
-model_type "bert-base-uncased" \
-task8_setting "1" \
-checkpoint "model_checkpoint_path (.pt)" &

  • Human evaluation metric for the three PASTA tasks are computed in ./code/human_eval.ipynb. Human response based on which these metrics are computed is in ./data/human_eval_dat.csv