Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add link_only option to remote file sources #19124

Open
guerler opened this issue Nov 9, 2024 · 0 comments
Open

Add link_only option to remote file sources #19124

guerler opened this issue Nov 9, 2024 · 0 comments

Comments

@guerler
Copy link
Contributor

guerler commented Nov 9, 2024

Suggesting to introduce an optional link_only boolean parameter in file_sources_conf.yml (default: false). When enabled, datasets are linked rather than copied during uploads. Although the upload/fetch mechanism already supports link_only uploads, this functionality is not currently configurable for file sources. By enabling this feature, users can avoid duplicating large datasets when the remote file source resides on a filesystem already mounted to their compute nodes. Potentially this can also be done using the file source type attribute.

Ping @jmchilton for suggestions and comments. Thanks in advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant