Skip to content

Latest commit

 

History

History
255 lines (223 loc) · 12.9 KB

README.md

File metadata and controls

255 lines (223 loc) · 12.9 KB
Raw

Quick tour - Debias DialoGPT by Transfer learning

As part of this thesis work DialoGPT is debiased for 5 demographics - Religion1 (Jews-Christians), Religion2 (Muslims-Christians), Race (African-American), Gender(Female-Male) and Sexual orientation (LGBTQ-Straight).

Below are the commands to carry out Algorithmic level and Data level Debiasing in pre-trained DialoGPT model. Examples are shown only for the demographic - Religion1 (Jews-Christains). In case of any other demographic, change the demographic fileds and data files accordingly.

Note: Debiasing scripts are found in the path debias_transformers/examples/language-modeling/. The data required all the below commands are found in https://github.com/SoumyaBarikeri/RedditBias/tree/master/data and https://github.com/SoumyaBarikeri/RedditBias/tree/master/text_files .

Note: The debiased models are found in dws-09 server at /work-ceph/sbariker/models/DEMOGRAPHIC_NAME/ -> Replace DEMOGRAPHIC_NAME with specific demographic like religion1, religion2, race, gender or orientation.

Algoritmic level Debiasing - Equalising loss over per sentence Target pairs

CUDA_VISIBLE_DEVICES=1 python debias_lm_grid.py \
    --output_dir=/work-ceph/sbariker/models/religion1/eq_loss_grid/ \
    --model_type=gpt2 \
    --model_name_or_path=microsoft/DialoGPT-small \
    --config_name=microsoft/DialoGPT-small \
    --tokenizer_name=microsoft/DialoGPT-small \
    --save_total_limit=2 \
    --num_train_epochs=2.0 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=2000 \
    --save_steps=2000 \
    --train_data_file=/work-ceph/sbariker/data/text_files/religion1/religion1_bias_manual_train.txt \
    --do_eval \
    --eval_data_file=/work-ceph/sbariker/data/text_files/humanref6k.txt \
    --block_size=36 \
    --line_by_line \
    --force_pad_token \
    --overwrite_output_dir \
    --debiasing_head=EqualisingLoss \
    --debias_method=EqualisingLoss \
    --embedding_type=output \
    --demographic=religion1 \
    --target_pair_type=per_sent_targets \
    --norm_debias_loss \
    --demo1_valid=/work-ceph/sbariker/data/text_files/religion1/religion1_jews_biased_valid_reduced.txt \
    --demo2_valid=/work-ceph/sbariker/data/text_files/religion1/religion1_christians_biased_valid_reduced.txt \
    --demo1_test=/work-ceph/sbariker/data/text_files/religion1/religion1_jews_biased_test_reduced.txt \
    --demo2_test=/work-ceph/sbariker/data/text_files/religion1/religion1_christians_biased_test_reduced.txt

Algoritmic level Debiasing - Cosine Distance equalising loss

CUDA_VISIBLE_DEVICES=1 python debias_lm_grid.py \
    --output_dir=/work-ceph/sbariker/models/religion1/cos_loss_grid/ \
    --model_type=gpt2 \
    --model_name_or_path=microsoft/DialoGPT-small \
    --config_name=microsoft/DialoGPT-small \
    --tokenizer_name=microsoft/DialoGPT-small \
    --save_total_limit=2 \
    --num_train_epochs=2.0 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=2000 \
    --save_steps=2000 \
    --train_data_file=/work-ceph/sbariker/data/text_files/religion1/religion1_bias_manual_train.txt \
    --do_eval \
    --eval_data_file=/work-ceph/sbariker/data/text_files/humanref6k.txt \
    --block_size=36 \
    --line_by_line \
    --force_pad_token \
    --overwrite_output_dir \
    --debiasing_head=Cosine \
    --debias_method=Cosine \
    --demographic=religion1 \
    --embedding_type=output \
    --demo1_valid=/work-ceph/sbariker/data/text_files/religion1/religion1_jews_biased_valid_reduced.txt \
    --demo2_valid=/work-ceph/sbariker/data/text_files/religion1/religion1_christians_biased_valid_reduced.txt \
    --demo1_test=/work-ceph/sbariker/data/text_files/religion1/religion1_jews_biased_test_reduced.txt \
    --demo2_test=/work-ceph/sbariker/data/text_files/religion1/religion1_christians_biased_test_reduced.txt

Algoritmic level Debiasing - Projection based Hard debiasing loss

CUDA_VISIBLE_DEVICES=1 python debias_lm_grid.py \
    --output_dir=/work-ceph/sbariker/models/religion1/hard_de_grid/ \
    --model_type=gpt2 \
    --model_name_or_path=microsoft/DialoGPT-small \
    --config_name=microsoft/DialoGPT-small \
    --tokenizer_name=microsoft/DialoGPT-small \
    --save_total_limit=2 \
    --num_train_epochs=2.0 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=2000 \
    --save_steps=2000 \
    --train_data_file=/work-ceph/sbariker/data/text_files/religion1/religion1_bias_manual_train.txt \
    --do_eval \
    --eval_data_file=/work-ceph/sbariker/data/text_files/humanref6k.txt \
    --block_size=36 \
    --line_by_line \
    --force_pad_token \
    --overwrite_output_dir \
    --debiasing_head=HardDe \
    --debias_method=HardDe \
    --demographic=religion1 \
    --embedding_type=output \
    --demo1_valid=/work-ceph/sbariker/data/text_files/religion1/religion1_jews_biased_valid_reduced.txt \
    --demo2_valid=/work-ceph/sbariker/data/text_files/religion1/religion1_christians_biased_valid_reduced.txt \
    --demo1_test=/work-ceph/sbariker/data/text_files/religion1/religion1_jews_biased_test_reduced.txt \
    --demo2_test=/work-ceph/sbariker/data/text_files/religion1/religion1_christians_biased_test_reduced.txt

Data level debiasing - Counter Target Data Augmentation (CTDA)

CUDA_VISIBLE_DEVICES=3 python run_language_modeling.py \
    --output_dir=/work-ceph/sbariker/models/religion1/lm_loss_swapped_target/ \
    --model_type=gpt2 \
    --model_name_or_path=microsoft/DialoGPT-small \
    --config_name=microsoft/DialoGPT-small \
    --tokenizer_name=microsoft/DialoGPT-small \
    --save_total_limit=2 \
    --num_train_epochs=2.0 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=2000 \
    --save_steps=2000 \
    --train_data_file=/work-ceph/sbariker/data/text_files/religion1/religion1_bias_manual_swapped_targets_train.txt \
    --do_eval \
    --eval_data_file=/work-ceph/sbariker/data/text_files/humanref6k.txt \
    --per_device_train_batch_size=2 \
    --per_device_eval_batch_size=2 \
    --block_size=36 \
    --gradient_accumulation_steps=1 \
    --line_by_line \
    --force_pad_token \
    --overwrite_output_dir

Evaluation: Significance results on testset -

python debias_transformers/evaluation/measure_bias_reduced_args.py     --data_path=/work-ceph/sbariker/data/     --log_path=/work-ceph/sbariker/logs/     --get_perp=yes     --save_perp=no     --demo=religion1     --demo1=jews     --demo2=christians     --input_file_1=reddit_comments_religion1_jews_processed_phrase_biased_testset_reduced.csv     --input_file_2=reddit_comments_religion1_christians_processed_phrase_biased_testset_reduced.csv     --model_path=/work-ceph/sbariker/models/religion1/lm_loss_swapped_target/    --model_name=lm_loss_swapped_target

Data level debiasing - Counter Attribute Data Augmentation (CADA)

CUDA_VISIBLE_DEVICES=0 python run_language_modeling.py \
    --output_dir=/work-ceph/sbariker/models/religion1/lm_loss_swapped_attr/ \
    --model_type=gpt2 \
    --model_name_or_path=microsoft/DialoGPT-small \
    --config_name=microsoft/DialoGPT-small \
    --tokenizer_name=microsoft/DialoGPT-small \
    --save_total_limit=2 \
    --num_train_epochs=2.0 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=2000 \
    --save_steps=2000 \
    --train_data_file=/work-ceph/sbariker/data/text_files/religion1/religion1_bias_manual_swapped_attr_train.txt \
    --do_eval \
    --eval_data_file=/work-ceph/sbariker/data/text_files/humanref6k.txt \
    --per_device_train_batch_size=2 \
    --per_device_eval_batch_size=2 \
    --block_size=36 \
    --gradient_accumulation_steps=1 \
    --line_by_line \
    --force_pad_token \
    --overwrite_output_dir

Evaluation: Significance results on testset -

python debias_transformers/evaluation/measure_bias_reduced_args.py     --data_path=/work-ceph/sbariker/data/     --log_path=/work-ceph/sbariker/logs/     --get_perp=yes     --save_perp=no     --demo=religion1     --demo1=jews     --demo2=christians     --input_file_1=reddit_comments_religion1_jews_processed_phrase_biased_testset_neg_attr_reduced.csv     --input_file_2=reddit_comments_religion1_jews_processed_phrase_unbiased_testset_pos_attr_reduced.csv     --model_path=/work-ceph/sbariker/models/religion1/lm_loss_swapped_attr/    --model_name=lm_loss_swapped_attr

Quick tour - Evaluation of Debiased models on Dialog State Tracking (DST) task

Below command evaluates DialoGPT debiased on Demographic - Religion1, based on Equalising loss

Note: The cleaned MultiWoz2 data can be found at https://github.com/SoumyaBarikeri/RedditBias/tree/master/data/clean_data . The Script (lm_dst_binary.py) to fine-tune models on DST task is found in debias_transformers/examples/language-modeling

Note: The debiased models fine-tuned on DST are found in /work-ceph/sbariker/models/dst/ on dws-09 server

CUDA_VISIBLE_DEVICES=0 python lm_dst_binary.py \
    --output_dir=/work-ceph/sbariker/models/dst/rel1_eq/ \
    --model_type=gpt2 \
    --model_name_or_path=/work-ceph/sbariker/models/religion1/eq_loss_grid/ \
    --config_name=/work-ceph/sbariker/models/religion1/eq_loss_grid/ \
    --tokenizer_name=/work-ceph/sbariker/models/religion1/eq_loss_grid/ \
    --save_total_limit=2 \
    --num_train_epochs=1.0 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=10000 \
    --save_steps=10000 \
    --train_data_file=/work-ceph/sbariker/data/multiwoz/clean_data/train_dials.json \
    --do_eval \
    --eval_data_file=/work-ceph/sbariker/data/multiwoz/clean_data/test_dials.json \
    --onto_file_path=/work-ceph/sbariker/data/multiwoz/ontology.json \
    --per_device_train_batch_size=12 \
    --per_device_eval_batch_size=12 \
    --block_size=128 \
    --gradient_accumulation_steps=4 \
    --line_by_line \
    --force_pad_token \
    --overwrite_output_dir \
    --label_names=dst_labels

Quick tour - Evaluation of Debiased models on Dialog System Technology Challenge 7 (DSTC7) Response generation task

Below command evaluates response generation capability of DialoGPT debiased on Demographic - Religion1, based on Equalising loss

Note: The Script (lm_dstc7.py) to fine-tune models on DSTC 7 task is found in debias_transformers/examples/language-modeling. The training dataset train_convos.txt for DSTC7 can be found in dws-09 university server at /work-ceph/sbariker/DSTC7-End-to-End-Conversation-Modeling/data_extraction/data-official/ , additionally the train_convos.txt can be generated referring to DSTC7 github (https://github.com/mgalley/DSTC7-End-to-End-Conversation-Modeling/tree/master/data_extraction). The test set can be found at https://github.com/SoumyaBarikeri/RedditBias/tree/master/data/dstc7 or /work-ceph/sbariker/DSTC7-End-to-End-Conversation-Modeling/data_extraction/data-official-test/test_convos_processed.txt .

Note: The debiased models fine-tuned on DSTC7 data are found in /work-ceph/sbariker/models/dstc7/ on dws-09 server

CUDA_VISIBLE_DEVICES=1 python lm_dstc7.py \
    --output_dir=/work-ceph/sbariker/models/dstc7/rel1_eq/ \
    --model_type=gpt2 \
    --model_name_or_path=/work-ceph/sbariker/models/religion1/eq_loss_grid/ \
    --config_name=/work-ceph/sbariker/models/religion1/eq_loss_grid/ \
    --tokenizer_name=/work-ceph/sbariker/models/religion1/eq_loss_grid/ \
    --save_total_limit=2 \
    --num_train_epochs=1 \
    --do_train \
    --evaluate_during_training \
    --logging_steps=10000 \
    --save_steps=10000 \
    --train_data_file=/work-ceph/sbariker/DSTC7-End-to-End-Conversation-Modeling/data_extraction/data-official/train_convos.txt \
    --do_eval \
    --eval_data_file=/work-ceph/sbariker/DSTC7-End-to-End-Conversation-Modeling/data_extraction/data-official-test/test_convos_processed.txt \
    --per_device_train_batch_size=16 \
    --per_device_eval_batch_size=16 \
    --gradient_accumulation_steps=5 \
    --line_by_line \
    --force_pad_token \
    --overwrite_output_dir \
    --output_resp_file=/work-ceph/sbariker/data/eval_dsct7/rel1_eq_resp.txt

Evaluation:

1. Generate test_convos.txt like file with '__UNDISCLOSED__' replaced with the model response

python debias_transformers/evaluation/prepare_dstc7_response.py --hyp_file=/work-ceph/sbariker/data/eval_dsct7/rel1_eq_resp.txt    --ref_file=/work-ceph/sbariker/DSTC7-End-to-End-Conversation-Modeling/data_extraction/data-official-test/test_convos.txt	--dest_file=/work-ceph/sbariker/data/eval_dsct7/rel1_eq_resp_test_convos.txt

2. Evaluate generated responses using dstc.py script provided by DSTC 7 team

Note: Clone the DSTC 7 repository and run the below command. Also the DSTC 7 data should be generated beforehand.

python dstc.py -c /work-ceph/sbariker/data/eval_dsct7/rel1_eq_resp_test_convos.txt --refs ../../data_extraction/test.refs