Teaching LLMs to do arithmetic tasks

Data description: In data file, we have:

bal is balanced addition dataset. It can be used to train addition and test addition.
mixed is mixed dataset. The expressions are different from bal. It can be used to train mixed tasks.
bal2 is another balanced addition dataset, which is not overlap with bal and addition_judge.
addition_judge is balanced judgment dataset. They can be used to trian judgment models. And test the ability of judgment.
- train_3digit_judge_10000.txt and test_3_digit_jugde_10000.txt are transfered from bal.
- train_3digit_judge_2_10000.txt and test_3_digit_jugde_2_10000.txt are transfered from bal2. It can be used to fine-tune a model that is pre-trained on bal.

Name		Name	Last commit message	Last commit date
Latest commit History 45 Commits
.vscode		.vscode
__pycache__		__pycache__
config/babygpt		config/babygpt
data		data
eval_checkpoints		eval_checkpoints
test_data		test_data
.gitignore		.gitignore
README.md		README.md
bench.py		bench.py
configurator.py		configurator.py
inference.ipynb		inference.ipynb
main_utils.py		main_utils.py
mask_others.ipynb		mask_others.ipynb
mean_para.ipynb		mean_para.ipynb
model.py		model.py
myscripts.sh		myscripts.sh
prompt.txt		prompt.txt
sample.py		sample.py
samples.txt		samples.txt
scaling_laws.ipynb		scaling_laws.ipynb
test.ipynb		test.ipynb
test_addition.ipynb		test_addition.ipynb
test_all.sh		test_all.sh
test_judge.ipynb		test_judge.ipynb
test_judge_ability.ipynb		test_judge_ability.ipynb
test_newdata.ipynb		test_newdata.ipynb
test_utils.py		test_utils.py
train.py		train.py
train_addition_eval.sh		train_addition_eval.sh
train_addition_mix.sh		train_addition_mix.sh
train_addition_plain.sh		train_addition_plain.sh
train_from_mistake.sh		train_from_mistake.sh
train_judge.sh		train_judge.sh
train_judge_from_add.sh		train_judge_from_add.sh
transformer_sizing.ipynb		transformer_sizing.ipynb

Provide feedback