Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Amberish runs #629

Merged
merged 214 commits into from
Jul 25, 2024
Merged

Amberish runs #629

merged 214 commits into from
Jul 25, 2024

Commits on May 13, 2024

  1. Adds a config that runs the whole thing in fp32

    @dirkgr
    dirkgr committed May 13, 2024
    Configuration menu
    Copy the full SHA
    9cd5127 View commit details
    Browse the repository at this point in the history

  2. Let's try running this on Jupiter

    @dirkgr
    dirkgr committed May 13, 2024
    Configuration menu
    Copy the full SHA
    fa403d8 View commit details
    Browse the repository at this point in the history

  3. Scripts should be executable

    @dirkgr
    dirkgr committed May 13, 2024
    Configuration menu
    Copy the full SHA
    e19ca62 View commit details
    Browse the repository at this point in the history

  4. There is no fp32 flash.

    @dirkgr
    dirkgr committed May 13, 2024
    Configuration menu
    Copy the full SHA
    6cd26f7 View commit details
    Browse the repository at this point in the history

  5. Needs more sharding

    @dirkgr
    dirkgr committed May 13, 2024
    Configuration menu
    Copy the full SHA
    87b3f0f View commit details
    Browse the repository at this point in the history

  6. change workspace

    @AkshitaB
    AkshitaB committed May 13, 2024
    Configuration menu
    Copy the full SHA
    18f0fee View commit details
    Browse the repository at this point in the history

  7. run on pluto

    @AkshitaB
    AkshitaB committed May 13, 2024
    Configuration menu
    Copy the full SHA
    a251fec View commit details
    Browse the repository at this point in the history

  8. eval frequently

    @AkshitaB
    AkshitaB committed May 13, 2024
    Configuration menu
    Copy the full SHA
    947b7d8 View commit details
    Browse the repository at this point in the history

  9. smaller microbatch

    @AkshitaB
    AkshitaB committed May 13, 2024
    Configuration menu
    Copy the full SHA
    ed77a49 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. fp32 llamaish from scratch

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    67adc77 View commit details
    Browse the repository at this point in the history

  2. training workspace

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    29cd968 View commit details
    Browse the repository at this point in the history

  3. use my secrets

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    0ba4121 View commit details
    Browse the repository at this point in the history

  4. run2

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    03d6ee6 View commit details
    Browse the repository at this point in the history

  5. use my secrets for run2

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    84516cc View commit details
    Browse the repository at this point in the history

  6. log grad norm each step

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    a915d67 View commit details
    Browse the repository at this point in the history

  7. run on pluto

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    742e572 View commit details
    Browse the repository at this point in the history

  8. no loading

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    02c2258 View commit details
    Browse the repository at this point in the history

  9. turn off flash attn

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    66a84ab View commit details
    Browse the repository at this point in the history

  10. change sharding

    @AkshitaB
    AkshitaB committed May 14, 2024
    Configuration menu
    Copy the full SHA
    32deea2 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. configs, launch scripts and llama2 tokenizer for training on pile

    @drschwenk
    drschwenk committed May 17, 2024
    Configuration menu
    Copy the full SHA
    a2fcaf3 View commit details
    Browse the repository at this point in the history

  2. changed run priority

    @drschwenk
    drschwenk committed May 17, 2024
    Configuration menu
    Copy the full SHA
    dc0ae92 View commit details
    Browse the repository at this point in the history

  3. change pad_token_id to match eos

    @drschwenk
    drschwenk committed May 17, 2024
    Configuration menu
    Copy the full SHA
    d6f79ad View commit details
    Browse the repository at this point in the history

  4. removed checkpoint load

    @drschwenk
    drschwenk committed May 17, 2024
    Configuration menu
    Copy the full SHA
    7842c2c View commit details
    Browse the repository at this point in the history

  5. turned off perplexity eval

    @drschwenk
    drschwenk committed May 17, 2024
    Configuration menu
    Copy the full SHA
    0b01c2d View commit details
    Browse the repository at this point in the history

  6. added load_path to resume run

    @drschwenk
    drschwenk committed May 17, 2024
    Configuration menu
    Copy the full SHA
    bf332dd View commit details
    Browse the repository at this point in the history

  7. specifying RDMA interface to use for NCCL

    @drschwenk
    drschwenk committed May 17, 2024
    Configuration menu
    Copy the full SHA
    767ba16 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. config for llm-360 amber data repro

    @drschwenk
    drschwenk committed May 28, 2024
    Configuration menu
    Copy the full SHA
    92c0e0c View commit details
    Browse the repository at this point in the history

  2. changed beaker secrets aws key/secret pointing to

    @drschwenk
    drschwenk committed May 28, 2024
    Configuration menu
    Copy the full SHA
    5ce8b70 View commit details
    Browse the repository at this point in the history

  3. change to pluto

    @drschwenk
    drschwenk committed May 28, 2024
    Configuration menu
    Copy the full SHA
    92695ba View commit details
    Browse the repository at this point in the history

  4. up node count, back to jupiter

    @drschwenk
    drschwenk committed May 28, 2024
    Configuration menu
    Copy the full SHA
    070dfac View commit details
    Browse the repository at this point in the history

  5. making job preemptible

    @drschwenk
    drschwenk committed May 28, 2024
    Configuration menu
    Copy the full SHA
    effc02a View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. initial config changes

    @drschwenk
    drschwenk committed May 29, 2024
    Configuration menu
    Copy the full SHA
    2e67908 View commit details
    Browse the repository at this point in the history

  2. change adamw eps to match llm-360

    @drschwenk
    drschwenk committed May 29, 2024
    Configuration menu
    Copy the full SHA
    1b1382c View commit details
    Browse the repository at this point in the history

  3. additional config changes

    @drschwenk
    drschwenk committed May 29, 2024
    Configuration menu
    Copy the full SHA
    cac419e View commit details
    Browse the repository at this point in the history

  4. change run name

    @drschwenk
    drschwenk committed May 29, 2024
    Configuration menu
    Copy the full SHA
    a160434 View commit details
    Browse the repository at this point in the history

  5. turn off qkv clipping

    @drschwenk
    drschwenk committed May 29, 2024
    Configuration menu
    Copy the full SHA
    2a80afb View commit details
    Browse the repository at this point in the history

  6. fix init_cutoff param

    @drschwenk
    drschwenk committed May 29, 2024
    Configuration menu
    Copy the full SHA
    61c87d3 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. fixing some issues/ matching a few more llm-360 hyperparams

    @drschwenk
    drschwenk committed May 31, 2024
    Configuration menu
    Copy the full SHA
    ea5061d View commit details
    Browse the repository at this point in the history

  2. removing overrides in the submission script

    @drschwenk
    drschwenk committed May 31, 2024
    Configuration menu
    Copy the full SHA
    ed15b17 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2024

  1. restart from last checkpoint

    @drschwenk
    drschwenk committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    3568d0e View commit details
    Browse the repository at this point in the history

  2. fix load_path

    @drschwenk
    drschwenk committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    6c71900 View commit details
    Browse the repository at this point in the history

  3. specify load_path

    @drschwenk
    drschwenk committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    9cfdb6f View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. update changes

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    63142d4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into epwalsh/amberish-stability

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    6ae2ff6 View commit details
    Browse the repository at this point in the history

  3. revert some other changes

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    037a51f View commit details
    Browse the repository at this point in the history

  4. prepare for launch

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    2d3e068 View commit details
    Browse the repository at this point in the history

  5. add w&B tag

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    bc5d9e7 View commit details
    Browse the repository at this point in the history

  6. change to urgent

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    96b7ee5 View commit details
    Browse the repository at this point in the history

  7. fixes

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    d77fca6 View commit details
    Browse the repository at this point in the history

  8. try without activation checkpointing

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b624e3e View commit details
    Browse the repository at this point in the history

  9. switch tokenizer

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    c4795a7 View commit details
    Browse the repository at this point in the history

  10. renames

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    db8c632 View commit details
    Browse the repository at this point in the history

  11. make executable

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    aefe6ec View commit details
    Browse the repository at this point in the history

  12. download HF cache

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    273636d View commit details
    Browse the repository at this point in the history

  13. Add amberish7 configs/scripts

    @epwalsh
    epwalsh committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    a4b11e7 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. load checkpoint

    @epwalsh
    epwalsh committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    324a9e6 View commit details
    Browse the repository at this point in the history

  2. no load

    @epwalsh
    epwalsh committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    a607ab6 View commit details
    Browse the repository at this point in the history

  3. run llm-360-amber-bz640

    @epwalsh
    epwalsh committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    1bd18c4 View commit details
    Browse the repository at this point in the history

  4. keep warmup still 2k steps

    @epwalsh
    epwalsh committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ad60997 View commit details
    Browse the repository at this point in the history

  5. update HF cache

    @epwalsh
    epwalsh committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    82be01e View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. restart amberish7-baseline

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    995bb31 View commit details
    Browse the repository at this point in the history

  2. run on more GPUs

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    686b644 View commit details
    Browse the repository at this point in the history

  3. try to debug slow speed

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    a89711b View commit details
    Browse the repository at this point in the history

  4. fix debug env var

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    09541e2 View commit details
    Browse the repository at this point in the history

  5. run with weka filesystem

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    fe19e1a View commit details
    Browse the repository at this point in the history

  6. turn on W&B again

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    0c9df12 View commit details
    Browse the repository at this point in the history

  7. try increasing micro batch size again

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    d83ba2f View commit details
    Browse the repository at this point in the history

  8. revert to 4

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    7860b89 View commit details
    Browse the repository at this point in the history

  9. increase number of GPUs for llm-360-amber-baseline

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    341cf85 View commit details
    Browse the repository at this point in the history

  10. fixes

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    395b323 View commit details
    Browse the repository at this point in the history

  11. fix

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    30b9a79 View commit details
    Browse the repository at this point in the history

  12. fix?

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    b50b8d9 View commit details
    Browse the repository at this point in the history

  13. save more often

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    295499a View commit details
    Browse the repository at this point in the history

  14. fix

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    7efcca9 View commit details
    Browse the repository at this point in the history

  15. start over

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    d3887f8 View commit details
    Browse the repository at this point in the history

  16. increase ephemeral save interval

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    8c03d7b View commit details
    Browse the repository at this point in the history

  17. restart from step 500

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    db0eaba View commit details
    Browse the repository at this point in the history

  18. ensure ranks know they are using shared fs

    @epwalsh
    epwalsh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    5ed3408 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2024

  1. try a different way

    @epwalsh
    epwalsh committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    bb4f034 View commit details
    Browse the repository at this point in the history

  2. load from latest

    @epwalsh
    epwalsh committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    6488ae9 View commit details
    Browse the repository at this point in the history

  3. run in debug

    @epwalsh
    epwalsh committed Jun 15, 2024
    Configuration menu
    Copy the full SHA
    4bb2e37 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2024

  1. restart for real

    @epwalsh
    epwalsh committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    526a346 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. add weight to RMS norm, add evals, use weka

    @epwalsh
    epwalsh committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    847f29b View commit details
    Browse the repository at this point in the history

  2. rename config

    @epwalsh
    epwalsh committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    40b7cf0 View commit details
    Browse the repository at this point in the history

  3. Make FSDP mixed precision optional

    @epwalsh
    epwalsh committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    5f5e956 View commit details
    Browse the repository at this point in the history

  4. turn off FSDP mixed precision

    @epwalsh
    epwalsh committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    16b1645 View commit details
    Browse the repository at this point in the history

  5. add activation checkpointing

    @epwalsh
    epwalsh committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    4c35bec View commit details
    Browse the repository at this point in the history

  6. run with different seed

    @epwalsh
    epwalsh committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    958a061 View commit details
    Browse the repository at this point in the history

  7. lol, fix

    @epwalsh
    epwalsh committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    6276645 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. increase warmup

    @epwalsh
    epwalsh committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    8545310 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. run with longer seq length

    @epwalsh
    epwalsh committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    dece9f0 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Add official Amberish configs

    @epwalsh
    epwalsh committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    9b3c1c4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into epwalsh/amberish

    @epwalsh
    epwalsh authored Jun 21, 2024
    Configuration menu
    Copy the full SHA
    4de8195 View commit details
    Browse the repository at this point in the history

  3. add ppl evaluations

    @epwalsh
    epwalsh committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    5f930e7 View commit details
    Browse the repository at this point in the history

  4. fix

    @epwalsh
    epwalsh committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    21dd46d View commit details
    Browse the repository at this point in the history

  5. propagate preemption

    @epwalsh
    epwalsh committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    1d34551 View commit details
    Browse the repository at this point in the history

  6. add load path

    @epwalsh
    epwalsh committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    867b4ed View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. update evals

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    9795374 View commit details
    Browse the repository at this point in the history

  2. tweak NCCL settings

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    abc8341 View commit details
    Browse the repository at this point in the history

  3. go to 512 GPUs

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    67c351b View commit details
    Browse the repository at this point in the history

  4. set timeout to 20

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    60114cb View commit details
    Browse the repository at this point in the history

  5. no timeout

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    d735d9f View commit details
    Browse the repository at this point in the history

  6. try different HCA setting

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    437abe0 View commit details
    Browse the repository at this point in the history

  7. try hybrid sharding

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    404de8b View commit details
    Browse the repository at this point in the history

  8. less replicas

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    ebad588 View commit details
    Browse the repository at this point in the history

  9. increase process group timeout

    @epwalsh
    epwalsh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    1e04e7d View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. decrease number of nodes

    @epwalsh
    epwalsh committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    23aa627 View commit details
    Browse the repository at this point in the history

  2. try NCCL_IB_GID_INDEX

    @epwalsh
    epwalsh committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    7d4d3b0 View commit details
    Browse the repository at this point in the history

  3. back to 8 nodes

    @epwalsh
    epwalsh committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    55aa293 View commit details
    Browse the repository at this point in the history

  4. prepare for 512 GPUs again

    @epwalsh
    epwalsh committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    224062d View commit details
    Browse the repository at this point in the history

  5. switch workspace

    @epwalsh
    epwalsh committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    4b3e5f2 View commit details
    Browse the repository at this point in the history

  6. revert back to old workspace

    @epwalsh
    epwalsh committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    3605e54 View commit details
    Browse the repository at this point in the history

  7. switch back to new workspace

    @epwalsh
    epwalsh committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    67a8bfb View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Fail fast if we fall back to ethernet

    @epwalsh
    epwalsh committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    a9d978c View commit details
    Browse the repository at this point in the history

  2. comment out unnecessary var

    @epwalsh
    epwalsh committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    fd13efd View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2024

  1. clean up

    @epwalsh
    epwalsh committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    c90a7ce View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. longer start timeout

    @epwalsh
    epwalsh committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    31e58a9 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. Add configs and launch scripts for Amberish1B

    @epwalsh
    epwalsh committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    59b0294 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. Merge branch 'main' into epwalsh/amberish

    @epwalsh
    epwalsh authored Jul 5, 2024
    Configuration menu
    Copy the full SHA
    238dd04 View commit details
    Browse the repository at this point in the history

  2. test run for 1B

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    1b89c52 View commit details
    Browse the repository at this point in the history

  3. try mbz of 4

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    e8c4f9d View commit details
    Browse the repository at this point in the history

  4. try mbz of 8

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    17506bd View commit details
    Browse the repository at this point in the history

  5. back to mbz of 4

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    01bdbd7 View commit details
    Browse the repository at this point in the history

  6. Add scripts for WD on embeddings

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    05cf130 View commit details
    Browse the repository at this point in the history

  7. start new epoch

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    94469ad View commit details
    Browse the repository at this point in the history

  8. Fix bug with epochs

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    363a316 View commit details
    Browse the repository at this point in the history

  9. restart

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    8730d46 View commit details
    Browse the repository at this point in the history

  10. fix

    @epwalsh
    epwalsh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    c215d66 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. go down to 8 nodes with 7B

    @epwalsh
    epwalsh committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    f7824bb View commit details
    Browse the repository at this point in the history

  2. set load path for 1Bs

    @epwalsh
    epwalsh committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    1434330 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. Merge branch 'main' into epwalsh/amberish

    @epwalsh
    epwalsh committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    553f323 View commit details
    Browse the repository at this point in the history

  2. Add scripts for 1B with selective updates

    @epwalsh
    epwalsh committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    5af5c6b View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Add unsharding scripts

    @epwalsh
    epwalsh committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    2776aab View commit details
    Browse the repository at this point in the history

  2. debug

    @epwalsh
    epwalsh committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    c81bfe3 View commit details
    Browse the repository at this point in the history

  3. revert

    @epwalsh
    epwalsh committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    d672805 View commit details
    Browse the repository at this point in the history

  4. copy to s3

    @epwalsh
    epwalsh committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    f99bc17 View commit details
    Browse the repository at this point in the history

  5. auto upload to S3

    @epwalsh
    epwalsh committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    6df4dbb View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. run on 16 nodes

    @epwalsh
    epwalsh committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    f595d06 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. pin flash attention

    @epwalsh
    epwalsh committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    7935cdb View commit details
    Browse the repository at this point in the history

  2. run selective updates without WD on embeddings

    @epwalsh
    epwalsh committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    737e113 View commit details
    Browse the repository at this point in the history

  3. Add options for emb init std and emb LN

    @epwalsh
    epwalsh committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    1fc4c21 View commit details
    Browse the repository at this point in the history

  4. Add scripts for amberish1-emb-ln

    @epwalsh
    epwalsh committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    cced92c View commit details
    Browse the repository at this point in the history

  5. use 8 nodes

    @epwalsh
    epwalsh committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    fb35549 View commit details
    Browse the repository at this point in the history

  6. use default std dev for embeddingds

    @epwalsh
    epwalsh committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e2ce48f View commit details
    Browse the repository at this point in the history

  7. try with just emb init 1

    @epwalsh
    epwalsh committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    c8db9f1 View commit details
    Browse the repository at this point in the history

  8. fix name

    @epwalsh
    epwalsh committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e9c4cfc View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2024

  1. load

    @epwalsh
    epwalsh committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    b3f1aeb View commit details
    Browse the repository at this point in the history

  2. add script with z-loss

    @epwalsh
    epwalsh committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    2cc3580 View commit details
    Browse the repository at this point in the history

  3. oops, fix

    @epwalsh
    epwalsh committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    ea86a79 View commit details
    Browse the repository at this point in the history

  4. try 2 replicas

    @epwalsh
    epwalsh committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    5118123 View commit details
    Browse the repository at this point in the history

  5. turn off hybrid

    @epwalsh
    epwalsh committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    b7a0161 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. Add 70B config and scripts

    @epwalsh
    epwalsh committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    c7bc32e View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Fix eos_token_id in configs

    @epwalsh
    epwalsh committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    279fbae View commit details
    Browse the repository at this point in the history

  2. update 70B config

    @epwalsh
    epwalsh committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    6186799 View commit details
    Browse the repository at this point in the history

  3. update

    @epwalsh
    epwalsh committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    a3916e4 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Add load path

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    bb2bf4e View commit details
    Browse the repository at this point in the history

  2. prep for run

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    4263354 View commit details
    Browse the repository at this point in the history

  3. adjust # of nodes

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    f1eb9bc View commit details
    Browse the repository at this point in the history

  4. no evals

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    dba79f7 View commit details
    Browse the repository at this point in the history

  5. match mitchish70 batch size and num nodes

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    b241565 View commit details
    Browse the repository at this point in the history

  6. rename

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    fe5073b View commit details
    Browse the repository at this point in the history

  7. tweak settings again

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    2832e71 View commit details
    Browse the repository at this point in the history

  8. prepare 7B

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    b1191ef View commit details
    Browse the repository at this point in the history

  9. prep for 256 GPUs

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    43dc713 View commit details
    Browse the repository at this point in the history

  10. more hybrid replicas

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    0eb2e04 View commit details
    Browse the repository at this point in the history

  11. try 16 replicas

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    279450f View commit details
    Browse the repository at this point in the history

  12. disable evaluators for now

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    22fc958 View commit details
    Browse the repository at this point in the history

  13. try no hybrid

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    52f4b79 View commit details
    Browse the repository at this point in the history

  14. run on 512

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    9e9454c View commit details
    Browse the repository at this point in the history

  15. 2 replicas

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    4ade8a2 View commit details
    Browse the repository at this point in the history

  16. update

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    22b81f1 View commit details
    Browse the repository at this point in the history

  17. 13B config

    @dirkgr
    dirkgr committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    0316e60 View commit details
    Browse the repository at this point in the history

  18. Merge branch 'epwalsh/amberish' of https://github.com/allenai/LLM int… 

    …o epwalsh/amberish
    @dirkgr
    dirkgr committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    7f5e812 View commit details
    Browse the repository at this point in the history

  19. big run

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    e3f0bf1 View commit details
    Browse the repository at this point in the history

  20. 4 replicas

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    25b44dc View commit details
    Browse the repository at this point in the history

  21. reset 7B config

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    a3cf646 View commit details
    Browse the repository at this point in the history

  22. prepare to run on 256 GPUs

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    b43447d View commit details
    Browse the repository at this point in the history

  23. turn off ephemeral checkpoints

    @epwalsh
    epwalsh committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    7a79e17 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. move scripts to dedicated folder

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    fb5372e View commit details
    Browse the repository at this point in the history

  2. fix merge conflicts

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    5e1c7ed View commit details
    Browse the repository at this point in the history

  3. Prep Amberish config with Chameleon fixes

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    cc86571 View commit details
    Browse the repository at this point in the history

  4. clean up

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    33150cf View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into epwalsh/amberish

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    4abe22a View commit details
    Browse the repository at this point in the history

  6. clean up

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    198fb5c View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into epwalsh/amberish

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    185cd1e View commit details
    Browse the repository at this point in the history

  8. clean up

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    5ba9d27 View commit details
    Browse the repository at this point in the history

  9. exclude norm reordering

    @epwalsh
    epwalsh committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    6e20f3e View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. tweaks

    @epwalsh
    epwalsh committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    fb811d9 View commit details
    Browse the repository at this point in the history

  2. update

    @epwalsh
    epwalsh committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    b35cd52 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into epwalsh/amberish

    @epwalsh
    epwalsh authored Jul 19, 2024
    Configuration menu
    Copy the full SHA
    3a4bfce View commit details
    Browse the repository at this point in the history

  4. clean up

    @epwalsh
    epwalsh committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    075b6e7 View commit details
    Browse the repository at this point in the history

  5. Add 8k context length scripts

    @epwalsh
    epwalsh committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    f68e29a View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into epwalsh/amberish

    @epwalsh
    epwalsh authored Jul 19, 2024
    Configuration menu
    Copy the full SHA
    525b315 View commit details
    Browse the repository at this point in the history

  7. try another chameleon

    @epwalsh
    epwalsh committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    12bf4cd View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2024

  1. run 8k with document masking

    @epwalsh
    epwalsh committed Jul 20, 2024
    Configuration menu
    Copy the full SHA
    ea724d4 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2024

  1. Add chameleon fixes to 8k run

    @epwalsh
    epwalsh committed Jul 21, 2024
    Configuration menu
    Copy the full SHA
    01f2bf6 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. prep 4k context length w/ doc masking

    @epwalsh
    epwalsh committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    5d53b6c View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. rename scripts

    @epwalsh
    epwalsh committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    71e9bf9 View commit details
    Browse the repository at this point in the history

  2. fix

    @epwalsh
    epwalsh committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    24d50c3 View commit details
    Browse the repository at this point in the history

  3. prep for 8k chameleon without doc masking

    @epwalsh
    epwalsh committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    2d0b8b0 View commit details
    Browse the repository at this point in the history

  4. use doc masking in ppl evals too

    @epwalsh
    epwalsh committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    95c477d View commit details
    Browse the repository at this point in the history

  5. fix

    @epwalsh
    epwalsh committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    64ba2a8 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. Make RoPE theta configurable, try a run w/ 500000

    @epwalsh
    epwalsh committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    b4e5e91 View commit details
    Browse the repository at this point in the history

  2. CHANGELOG

    @epwalsh
    epwalsh committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    b4acf5b View commit details
    Browse the repository at this point in the history

  3. add config with new data, new tokenizer

    @epwalsh
    epwalsh committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    00dfa78 View commit details
    Browse the repository at this point in the history

  4. prep peteish run

    @epwalsh
    epwalsh committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4ff8598 View commit details
    Browse the repository at this point in the history

  5. remove ppl sets for now

    @epwalsh
    epwalsh committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    d709d2d View commit details
    Browse the repository at this point in the history

  6. clean up unshard script

    @epwalsh
    epwalsh committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    ba67ac0 View commit details
    Browse the repository at this point in the history

  7. fix typo

    @epwalsh
    epwalsh committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    27fb675 View commit details
    Browse the repository at this point in the history

  8. Update olmo/config.py 

    Co-authored-by: Shane A <[email protected]>
    @epwalsh @2015aroras
    epwalsh and 2015aroras authored Jul 24, 2024
    Configuration menu
    Copy the full SHA
    ce930dd View commit details
    Browse the repository at this point in the history

  9. back to 16 nodes for now

    @epwalsh
    epwalsh committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    ac6b650 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Add peteish 7B config

    @epwalsh
    epwalsh committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1e32421 View commit details
    Browse the repository at this point in the history

  2. back to 4k

    @epwalsh
    epwalsh committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    accc3ee View commit details
    Browse the repository at this point in the history

  3. final tweaks

    @epwalsh
    epwalsh committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    8b7afd0 View commit details
    Browse the repository at this point in the history