Padding free squashing

[garrett361]

Squashes various commits and rebases on top of latest upstream main.

[fabianlim]

@garrett361 i didnt realize there is a quality check, its failing the check now. otherwise looks quite good to me

[fabianlim]

not sure if this is meaningful, since there are going to be many iterations on these templates. I feel its better to allow the template to be a file path, and it gets read in

[garrett361]

Divya required this for her work. Easy to drop this later since it's a standalone commit.

[fabianlim]

could we have a feature that we can set chat_template_name to reference a file, and that it could be read and set here. two options;

1. set chat_template_name="custom" and have a few flag chat_template_custom_file to point to the new file
2. overload chat_template_name to simply point to a new file.

[garrett361]

Yeah, I think I'll do 2: if chat_template_name isn't in CHAT_TEMPLATES, then try to read in the text from a file at path chat_template_name. Kinda hacky, but then we don't have to add more flags.

[fabianlim]

hmm.. but maybe 1. is better because its more consistent..

[garrett361]

@fabianlim also didn't realize about the check; guess it's only activated when PR-ing against main.

[garrett361]

Fixed the CI lint failures.

[fabianlim]

I think this will be a good to merge soon, although we need to document down what is missing in comparison to padding-free branch

1. accept a train_file input
2. parquet file support and additional model args Small Updates to Padding Free Branch #2
3. tools and documents (is this still needed)? Add Support For Tools and Documents Chat Template #3
4. dataloader tester Script for Data Loading Testing #7
5. add_column is slow so use pandas to load dataset https://github.com/garrett361/open-instruct/pull/6/files

[fabianlim]

this seems to be superceeded by the series of PRs #17 , etc.. so maybe we should close it?

[garrett361]

Closing, due to superceeding PRs.