fix: ulimit set in ray.sub

[bogdansalyp]

What does this PR do ?

Fixes a crash on a cluster with low ulimit

Issues

Closes #988

Usage

No changes to regular flow

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• continuing Stan's work at chore: ray-sub - improve robustness #968
• it is recommended to set ulimit in ray best practice docs
• ulimit is set in a session scope, so the changes made affect only the script runtime and not the system outside

[bogdansalyp]

@terrykong also do you know if that's a deliberate decision to have ray.sub -rw-r--r-- ? I'd make it chmod +x

[terrykong]

thanks for the fix @bogdansalyp

not deliberate, but is there a reason you want it to be executable? most of the sbatch args have to be provided based on diff users cluster env so the obvious saving is you don't need to specify sbatch if you put it in the #! header.

[bogdansalyp]

@terrykong makes sense, I've just tried some automation but realized there's an easier way to solve what I'm looking for
so chmod +x isn't important at the moment, but the ulimit thing is
pls approve if everything's fine with it

[terrykong]

tested internally. lgtm. thanks for making nemo-rl better!