feat(services/webhdfs): Implement multi write via CONCAT

[hoslo]

fixed #3924

[hoslo]

I run the behavior test locally and I succeed, what are the reasons for failure in CI?

[Xuanwo]

I run the behavior test locally and I succeed, what are the reasons for failure in CI?

Write failed for:

  2024-01-09T09:03:45.037494Z  WARN opendal::services: service=webhdfs operation=Writer::close path=3bfb3b38-eb71-4a3d-8699-774887cd431c written=10485760B -> data close failed: NotFound (permanent) at Writer::close, context: { uri: http://127.0.0.1:9870/webhdfs/v1/07b41151-d577-442f-8505-a6fd7bb6063e/3bfb3b38-eb71-4a3d-8699-774887cd431c?op=CONCAT&sources=/07b41151-d577-442f-8505-a6fd7bb6063e/da42e868-6285-4745-a88d-b2dd2d826eb2%2C/07b41151-d577-442f-8505-a6fd7bb6063e/92daa91e-9513-4165-8f8b-0d91fa97175b, response: Parts { status: 404, version: HTTP/1.1, headers: {"date": "Tue, 09 Jan 2024 09:03:45 GMT", "date": "Tue, 09 Jan 2024 09:03:45 GMT", "cache-control": "no-cache", "expires": "Tue, 09 Jan 2024 09:03:45 GMT", "pragma": "no-cache", "x-frame-options": "SAMEORIGIN", "content-type": "application/json", "transfer-encoding": "chunked"} }, service: webhdfs, path: 3bfb3b38-eb71-4a3d-8699-774887cd431c } => WebHdfsError { exception: "FileNotFoundException", message: "File does not exist: /07b41151-d577-442f-8505-a6fd7bb6063e/92daa91e-9513-4165-8f8b-0d91fa97175b", java_class_name: "java.io.FileNotFoundException" }

[hoslo]

I run the behavior test locally and I succeed, what are the reasons for failure in CI?

Write failed for:

  2024-01-09T09:03:45.037494Z  WARN opendal::services: service=webhdfs operation=Writer::close path=3bfb3b38-eb71-4a3d-8699-774887cd431c written=10485760B -> data close failed: NotFound (permanent) at Writer::close, context: { uri: http://127.0.0.1:9870/webhdfs/v1/07b41151-d577-442f-8505-a6fd7bb6063e/3bfb3b38-eb71-4a3d-8699-774887cd431c?op=CONCAT&sources=/07b41151-d577-442f-8505-a6fd7bb6063e/da42e868-6285-4745-a88d-b2dd2d826eb2%2C/07b41151-d577-442f-8505-a6fd7bb6063e/92daa91e-9513-4165-8f8b-0d91fa97175b, response: Parts { status: 404, version: HTTP/1.1, headers: {"date": "Tue, 09 Jan 2024 09:03:45 GMT", "date": "Tue, 09 Jan 2024 09:03:45 GMT", "cache-control": "no-cache", "expires": "Tue, 09 Jan 2024 09:03:45 GMT", "pragma": "no-cache", "x-frame-options": "SAMEORIGIN", "content-type": "application/json", "transfer-encoding": "chunked"} }, service: webhdfs, path: 3bfb3b38-eb71-4a3d-8699-774887cd431c } => WebHdfsError { exception: "FileNotFoundException", message: "File does not exist: /07b41151-d577-442f-8505-a6fd7bb6063e/92daa91e-9513-4165-8f8b-0d91fa97175b", java_class_name: "java.io.FileNotFoundException" }

I encountered this error, but it's not necessarily reproducible. Running the test that failed on its own doesn't show any issues.

[Xuanwo]

I encountered this error, but it's not necessarily reproducible. Running the test that failed on its own doesn't show any issues.

Seems the error here is not checked? You can fix block_write related staffs in this PR directly.

https://github.com/apache/incubator-opendal/blob/b956b76d5fa02c069516980df4b428c35fa49e49/core/src/raw/oio/write/block_write.rs#L235

[hoslo]

I encountered this error, but it's not necessarily reproducible. Running the test that failed on its own doesn't show any issues.

Seems the error here is not checked? You can fix block_write related staffs in this PR directly.

https://github.com/apache/incubator-opendal/blob/b956b76d5fa02c069516980df4b428c35fa49e49/core/src/raw/oio/write/block_write.rs#L235

Still have a problem.

[hoslo]

@Xuanwo The issue is caused by automatic redirection, now you can proceed with the merge.

[Xuanwo]

Thanks a lot!

[Xuanwo]

Thank you so much for your patience!