HADOOP-18330 S3AFileSystem removes Path when calling createS3Client

[ashutoshpant]

First of all, sorry for the multiple PR's, it's because i cant push from my device because of security reasons and have to use https://github.dev/

Description of PR

Added a new parameter object (pathUrl) that holds the full s3a path

How was this patch tested?

• Ran all the tests successfully using mvn clean compile package.
• Used the jar from above step to successfully read/write to an S3 bucket in us-east. Repeated this 3 times.

For code changes:

• [ X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.3.3 Compile Tests _
+1 💚	mvninstall	35m 19s		branch-3.3.3 passed
+1 💚	compile	0m 57s		branch-3.3.3 passed
+1 💚	checkstyle	0m 53s		branch-3.3.3 passed
+1 💚	mvnsite	1m 3s		branch-3.3.3 passed
+1 💚	javadoc	0m 56s		branch-3.3.3 passed
+1 💚	spotbugs	1m 46s		branch-3.3.3 passed
+1 💚	shadedclient	25m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 46s		the patch passed
+1 💚	compile	0m 38s		the patch passed
+1 💚	javac	0m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 28s		the patch passed
+1 💚	mvnsite	0m 45s		the patch passed
+1 💚	javadoc	0m 33s		the patch passed
+1 💚	spotbugs	1m 20s		the patch passed
+1 💚	shadedclient	24m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 42s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 52s		The patch does not generate ASF License warnings.
		106m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4557/1/artifact/out/Dockerfile
GITHUB PR	#4557
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 8143d2669d56 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3.3 / 74d5e3f
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4557/1/testReport/
Max. process+thread count	539 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4557/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[steveloughran]

code looks great.

there's a few places in the test code where a new S3ClientFactory.S3ClientCreationParameters() is created
and filled in. Can you set the path there for completeness?

[steveloughran]

changes look great; no more suggestions.

now, testing diligence.

Which aws region did you run the hadoop-aws integration tests against, what where the cli settings?

this isn't just to put homework on you, the more people testing with different configs, the more likely we are to find failures before they ship.

+1 pending those aws-test results

[ashutoshpant]

changes look great; no more suggestions.

now, testing diligence.

Which aws region did you run the hadoop-aws integration tests against, what where the cli settings?

this isn't just to put homework on you, the more people testing with different configs, the more likely we are to find failures before they ship.

+1 pending those aws-test results
AWS Region: us-east-1
IT test command: mvn -Dparallel-tests -DtestsThreadCount=8 clean verify
other test: mvn clean test
Settings used:

    <property>
        <name>test.fs.s3a.name</name>
        <value>s3a://test-hadoop1/</value>
    </property>

    <property>
        <name>fs.contract.test.fs.s3a</name>
        <value>${test.fs.s3a.name}</value>
    </property>

    <property>
        <name>fs.s3a.access.key</name>
        <description>AWS access key ID. Omit for IAM role-based authentication.</description>
        <value></value>
    </property>

    <property>
        <name>fs.s3a.secret.key</name>
        <description>AWS secret key. Omit for IAM role-based authentication.</description>
        <value></value>
    </property>

    <property>
        <name>fs.s3a.endpoint</name>
        <value>s3.us-east-1.amazonaws.com</value>
    </property>

    <property>
        <name>test.fs.s3a.encryption.enabled</name>
        <value>false</value>
    </property>

    <property>
        <name>test.fs.s3a.sts.enabled</name>
        <value>false</value>
    </property>

</configuration>

[steveloughran]

aah, I'd merged this and only then noticed this was against 3.3.3. reverted.

Can you create a PR with the final patch applied to trunk? and test it (just tell us the endpoint, no need for the other details). then we can merge there and back in to branch-3.3

the 3.3.3 branch is frozen; a fork was made earlier for a critical integration/cve release, which this doesn't qualify for...target the next release after that

[ashutoshpant]

aah, I'd merged this and only then noticed this was against 3.3.3. reverted.

Can you create a PR with the final patch applied to trunk? and test it (just tell us the endpoint, no need for the other details). then we can merge there and back in to branch-3.3

the 3.3.3 branch is frozen; a fork was made earlier for a critical integration/cve release, which this doesn't qualify for...target the next release after that

Oops!! sorry! Just created a new PR: #4572