Add support for NeMo SDK #131
Conversation
Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Ryan Wolf <[email protected]>
| interface: str = "eth0" | ||
| protocol: str = "tcp" | ||
| cpu_worker_memory_limit: str = "0" | ||
| rapids_no_initialize: str = "1" | ||
| cudf_spill: str = "1" | ||
| rmm_scheduler_pool_size: str = "1GB" | ||
| rmm_worker_pool_size: str = "72GiB" | ||
| libcudf_cufile_policy: str = "OFF" |
There was a problem hiding this comment.
Similar to my other comment, I often have trouble knowing what to set for these types of parameters. Is there anywhere the user might be able to refer to for recommendations of how to set these parameters for their specific cluster?
There was a problem hiding this comment.
I agree we should release a bigger guide on our recommendations for each parameter. For now I've included a docstring that should provide a bit more context. Let me know if you want me to change anything else to make it clearer.
Signed-off-by: Ryan Wolf <[email protected]>
| # Path to NeMo-Curator/examples/slurm/container_entrypoint.sh on the SLURM cluster | ||
| container_entrypoint = "/cluster/path/slurm/container_entrypoint.sh" | ||
| # The NeMo Curator command to run | ||
| curator_command = "text_cleaning --input-data-dir=/path/to/data --output-clean-dir=/path/to/output" |
There was a problem hiding this comment.
Can this work for multiple commands ?
There was a problem hiding this comment.
Yes, this should be able to work for all dask-based curator scripts. It'll work for the rest too, though it'll be a bit overkill.
There was a problem hiding this comment.
Cool , we can leave it as is. Maybe we add two.
| mkdir -p $LOGDIR | ||
| mkdir -p $PROFILESDIR |
There was a problem hiding this comment.
This is run inside the container ?
There was a problem hiding this comment.
Yes you're correct. It's a good call out, and it makes me think we maybe should've done this from the beginning since the contents of $LOGDIR and $PROFILESDIR get written inside the container, so they ought to be initialized in it too. Let me know if you disagree.
There was a problem hiding this comment.
So, i think because logs are accessed from outside the container so we should make it clear where the path is from outside the container, I think to make it clear we should echo $LOGDIR and $PROFILESDIR to help someone debugging this.
Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Ryan Wolf <[email protected]>
Signed-off-by: Ryan Wolf <[email protected]>
Description
NeMo SDK is a library designed to make running different parts of the NeMo FW easier across computing platforms. It serves as an enhanced version of the NeMo Framework Launcher. This PR adds an example and simple config shortcut to run NeMo Curator scripts on Slurm clusters using NeMo SDK.
Usage
See
examples/nemo_sdk/slurm.pyfor example usage.Checklist