Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loss go NAN when running classification model of Seaformer_B #2

Open
shiyutang opened this issue Feb 7, 2023 · 6 comments
Open

Loss go NAN when running classification model of Seaformer_B #2

shiyutang opened this issue Feb 7, 2023 · 6 comments

Comments

@shiyutang
Copy link

shiyutang commented Feb 7, 2023

I run the following command as described in the readme, but the loss goes NAN. I wonder why this happens.
image

@wwqq
Copy link
Collaborator

wwqq commented Feb 7, 2023

Try to train the model with 8 gpus.

@shiyutang
Copy link
Author

Thanks a lot, I must forget to change to 8 after test.

@1787648106
Copy link

hi,I trained the model with 8 gpus, but also goes nan.
image
The --resume arg is not specified in the startup command, will this not affect it?

@1787648106
Copy link

Thanks a lot, I must forget to change to 8 after test.

Have you solved this problem?

@shiyutang
Copy link
Author

Yes, It needs to be trained with 8 gpus.

@1787648106
Copy link

Yes, It needs to be trained with 8 gpus.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants