CodeParrot data pretokenization#16932

Merged

loubnabnl merged 18 commits intohuggingface:mainfrom

loubnabnl:codeparrot-data-pretokenization

May 16, 2022

Commits on May 4, 2022

add pretokenization arguments

loubnabnl
authored and
Loubna ben allal
committed
add pretokenization script

loubnabnl
authored and
Loubna ben allal
committed
add support for pretokenized data

loubnabnl
authored and
Loubna ben allal
committed
reformat code

loubnabnl
authored and
Loubna ben allal
committed
fix run command for training

loubnabnl
authored and
Loubna ben allal
committed
fix model call from config

loubnabnl
authored and
Loubna ben allal
committed
remove a package

loubnabnl
authored and
Loubna ben allal
committed
add comments on pretokenization in the readme

loubnabnl
authored and
Loubna ben allal
committed
remove explicit parallelization

authored and
Loubna ben allal
committed
update readme

authored and
Loubna ben allal
committed
update readme -remove username

authored and
Loubna ben allal
committed
update readme -remove username

authored and
Loubna ben allal
committed
keep data parallelization
Loubna ben allal
committed
reformat code
Loubna ben allal
committed
reformat code
Loubna ben allal
committed
update readme
Loubna ben allal
committed
reformat code
Loubna ben allal
committed

Commits on May 16, 2022

Update examples/research_projects/codeparrot/README.md

loubnabnl
and
lvwerra
authored