Add a unified remote I/O interface that infers the endpoint type from URL (1/2): C++ implementation by kingcrimsontianyu · Pull Request #793 · rapidsai/kvikio

kingcrimsontianyu · 2025-08-07T05:44:00Z

This PR adds a new remote I/O utility function RemoteHandle::open(url) that infers the remote endpoint type from the URL to facilitate RemoteHandle creation.

Supported endpoint types include S3, S3 with presigned URL, WebHDFS, and generic HTTP/HTTPS.
Optionally, instead of letting open figure it out, users can explicitly specify the endpoint type by passing an enum argument RemoteEndpointType.
Optionally, users can provide an allowlist that restricts the endpoint candidates
Optionally, users can specify the expected file size. This design is to fully support the existing constructor overload RemoteHandle(endpoint, nbytes).

A byproduct of this PR is an internal utility class UrlParser that uses the idiomatic libcurl URL API to validate the URL against "RFC 3986 plus".

This PR depends on

…rrent PR

copy-pr-bot · 2025-08-07T05:44:03Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…destroy the remote file only once per test suite instead of per test

Co-authored-by: Mads R. B. Kristensen <madsbk@gmail.com>

kingcrimsontianyu · 2025-08-22T20:09:19Z

/ok to test 64e8713

kingcrimsontianyu · 2025-08-22T20:25:40Z

/ok to test 68e3a16

kingcrimsontianyu · 2025-08-23T04:58:55Z

An issue not addressed in this PR and requires more thinking in the future:

Endpoints on the current 25.10 only perform light syntax check for URL using regular expressions. Should we add more extensive URL validation (RFC 3986 plus) inside the constructor, or should we keep validation and construction separated?

In this PR, the open function explicitly calls is_url_valid to check the URL validity against a certain endpoint, and if valid, proceeds to instantiate the endpoint. There would be duplicate validation if we add it to the constructor. Should we simply tolerate an ABI change to add an optional (and perhaps unsightly) bool validate_url=true/false parameter to the endpoint constructor?

kingcrimsontianyu · 2025-08-23T05:00:29Z

/ok to test a498a83

cpp/include/kvikio/remote_handle.hpp

cpp/src/detail/url.cpp

cpp/src/remote_handle.cpp

kingcrimsontianyu · 2025-08-25T03:14:17Z

/ok to test bba7253

madsbk

Looks good, I only have minor suggestions

cpp/include/kvikio/detail/url.hpp

cpp/include/kvikio/remote_handle.hpp

cpp/src/remote_handle.cpp

kingcrimsontianyu · 2025-08-25T13:20:24Z

/ok to test 357a615

kingcrimsontianyu · 2025-08-25T16:03:44Z

/merge

… URL (2/2): Python binding (#808) This PR adds Python binding to #793 Closes #807 Authors: - Tianyu Liu (https://github.com/kingcrimsontianyu) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #808

With rapidsai/kvikio#793, KvikIO can infer the endpoint type from the URL, supporting creation of a broader range of remote resources via a single interface. This PR updates the cuDF data source accordingly. Specifically, `pylibcudf` now can read from WebHDFS, S3, S3 presigned URL resources. Partially addresses #19633 Authors: - Tianyu Liu (https://github.com/kingcrimsontianyu) - Vukasin Milovanovic (https://github.com/vuule) - Matthew Murray (https://github.com/Matt711) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Nghia Truong (https://github.com/ttnghia) - Muhammad Haseeb (https://github.com/mhaseeb123) - Vukasin Milovanovic (https://github.com/vuule) - Matthew Murray (https://github.com/Matt711) URL: #19788

kingcrimsontianyu and others added 16 commits August 5, 2025 03:29

Support WebHDFS: C++ implementation

d406591

Update

2e8328f

Move advanced URL handling to a separate PR to reduce the scope of cu…

46c1cbb

…rrent PR

Update

8fd6a70

Add clarifying comments

cecd49c

Add more comments

e6e4d56

Update

97da554

Update

6b48777

Add default arg

e4fbd35

Add Python binding for WebHDFS

3792549

Fix a bug where too large file causes string-to-size conversion to fail

87603f0

Merge branch 'web-hdfs' into python-web-hdfs

95e1c91

Merge branch 'branch-25.10' into web-hdfs

c3881e0

Merge branch 'web-hdfs' into python-web-hdfs

c91fcfb

Unified interface for various endpoint types

cef9556

Update

e659c2f

kingcrimsontianyu added improvement Improves an existing functionality non-breaking Introduces a non-breaking change c++ Affects the C++ API of KvikIO labels Aug 7, 2025

kingcrimsontianyu added the DO NOT MERGE label Aug 7, 2025

kingcrimsontianyu and others added 8 commits August 7, 2025 10:05

For WebHDFS unit test, fix segfault when the test is skipped; create/…

bb256f2

…destroy the remote file only once per test suite instead of per test

Remove unnecessary ntvx annotation

969f6d0

Update cpp/tests/CMakeLists.txt

377d88e

Co-authored-by: Mads R. B. Kristensen <madsbk@gmail.com>

Attempt to fix CI overlinking error

9c6477c

Merge branch 'web-hdfs' into python-web-hdfs

fee32db

Remove libcurl from libkvikio-tests' run section

0507d78

Merge branch 'web-hdfs' into python-web-hdfs

ec0871e

Merge branch 'python-web-hdfs' into remote-io-easy-interface

59c683d

kingcrimsontianyu changed the title ~~Add a unified remote I/O interface that infers the endpoint type from URL~~ Add a unified remote I/O interface that infers the endpoint type from URL (1/2): C++ implementation Aug 7, 2025

Improve test and impl

64e8713

kingcrimsontianyu removed the DO NOT MERGE label Aug 22, 2025

Add more comments. Improve test

68e3a16

kingcrimsontianyu added 4 commits August 22, 2025 23:57

Add comments to the unified open function

b5dbcbb

Improve doxygen comment

2b526e2

Improve comment

d44e165

Update doc

0d3ab0b

kingcrimsontianyu marked this pull request as ready for review August 23, 2025 04:26

kingcrimsontianyu requested review from a team as code owners August 23, 2025 04:26

Fix doc inaccuracy

a498a83

rapidsai deleted a comment from copy-pr-bot bot Aug 23, 2025

madsbk reviewed Aug 24, 2025

View reviewed changes

Address review comments

bba7253

kingcrimsontianyu requested a review from madsbk August 25, 2025 03:14

madsbk approved these changes Aug 25, 2025

View reviewed changes

cpp/include/kvikio/detail/url.hpp Outdated Show resolved Hide resolved

cpp/include/kvikio/detail/url.hpp Outdated Show resolved Hide resolved

cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved

cpp/src/remote_handle.cpp Outdated Show resolved Hide resolved

Address review comments

357a615

rapids-bot bot merged commit 46fa7dd into rapidsai:branch-25.10 Aug 25, 2025
77 checks passed

Comments

Conversation

kingcrimsontianyu commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR depends on

Uh oh!

copy-pr-bot bot commented Aug 7, 2025

Uh oh!

kingcrimsontianyu commented Aug 22, 2025

Uh oh!

kingcrimsontianyu commented Aug 22, 2025

Uh oh!

kingcrimsontianyu commented Aug 23, 2025

Uh oh!

kingcrimsontianyu commented Aug 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kingcrimsontianyu commented Aug 25, 2025

Uh oh!

madsbk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kingcrimsontianyu commented Aug 25, 2025

Uh oh!

kingcrimsontianyu commented Aug 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kingcrimsontianyu commented Aug 7, 2025 •

edited

Loading