Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Amberish runs #629

Merged
merged 214 commits into from
Jul 25, 2024
Merged

Amberish runs #629

merged 214 commits into from
Jul 25, 2024

Conversation

epwalsh
Copy link
Member

@epwalsh epwalsh commented Jun 21, 2024 •
edited
Loading

Includes some additions to the model config.

dirkgr and others added 30 commits May 12, 2024 22:52
@dirkgr
Adds a config that runs the whole thing in fp32
9cd5127
@dirkgr
Let's try running this on Jupiter
fa403d8
@dirkgr
Scripts should be executable
e19ca62
@dirkgr
There is no fp32 flash.
6cd26f7
@dirkgr
Needs more sharding
87b3f0f
@AkshitaB
change workspace
18f0fee
@AkshitaB
run on pluto
a251fec
@AkshitaB
eval frequently
947b7d8
@AkshitaB
smaller microbatch
ed77a49
@AkshitaB
fp32 llamaish from scratch
67adc77
@AkshitaB
training workspace
29cd968
@AkshitaB
use my secrets
0ba4121
@AkshitaB
run2
03d6ee6
@AkshitaB
use my secrets for run2
84516cc
@AkshitaB
log grad norm each step
a915d67
@AkshitaB
run on pluto
742e572
@AkshitaB
no loading
02c2258
@AkshitaB
turn off flash attn
66a84ab
@AkshitaB
change sharding
32deea2
@drschwenk
configs, launch scripts and llama2 tokenizer for training on pile
a2fcaf3
@drschwenk
changed run priority
dc0ae92
@drschwenk
change pad_token_id to match eos
d6f79ad
@drschwenk
removed checkpoint load
7842c2c
@drschwenk
turned off perplexity eval
0b01c2d
@drschwenk
added load_path to resume run
bf332dd
@drschwenk
specifying RDMA interface to use for NCCL
767ba16
@drschwenk
config for llm-360 amber data repro
92c0e0c
@drschwenk
changed beaker secrets aws key/secret pointing to
5ce8b70
@drschwenk
change to pluto
92695ba
@drschwenk
up node count, back to jupiter
070dfac
@epwalsh epwalsh changed the title Amberish 7B hero run Amberish runs Jul 24, 2024
@epwalsh epwalsh marked this pull request as ready for review July 24, 2024 18:41
soldni
soldni approved these changes Jul 24, 2024
View reviewed changes
Copy link
Member

@soldni soldni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

2015aroras
2015aroras approved these changes Jul 24, 2024
View reviewed changes
configs/amberish13-weka.yaml Outdated Show resolved Hide resolved
olmo/config.py Outdated Show resolved Hide resolved
scripts/beaker/unshard.sh Outdated Show resolved Hide resolved
scripts/beaker/unshard.sh Outdated Show resolved Hide resolved
@epwalsh epwalsh merged commit 3e30710 into main Jul 25, 2024
12 checks passed
@epwalsh epwalsh deleted the epwalsh/amberish branch July 25, 2024 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@2015aroras 2015aroras 2015aroras approved these changes

@soldni soldni soldni approved these changes

@AkshitaB AkshitaB Awaiting requested review from AkshitaB

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@epwalsh @soldni @2015aroras @dirkgr @AkshitaB @drschwenk