[SPARK-31767][PYTHON][CORE] Remove ResourceInformation in pyspark module's namespace

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to only allow the import of ResourceInformation as below:

pyspark.resource.ResourceInformation

instead of

pyspark.ResourceInformation
pyspark.resource.ResourceInformation

because pyspark.resource is a separate module, and it is documented so.
The constructor of ResourceInformation isn't supposed to directly call anyway.

Why are the changes needed?

To keep the code structure coherent.

Does this PR introduce any user-facing change?

No, it will be in the unreleased branches.

How was this patch tested?

Manually tested via importing:

Before:

>>> import pyspark
>>> pyspark.ResourceInformation
<class 'pyspark.resource.information.ResourceInformation'>
>>> pyspark.resource.ResourceInformation
<class 'pyspark.resource.information.ResourceInformation'>

After:

>>> import pyspark
>>> pyspark.ResourceInformation
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'pyspark' has no attribute 'ResourceInformation'
>>> pyspark.resource.ResourceInformation
<class 'pyspark.resource.information.ResourceInformation'>

Also tested via

cd python
./run-tests --python-executables=python3 --modules=pyspark-core,pyspark-resource

Jenkins will test and existing tests should cover.

[HyukjinKwon]

This can be backported to branch-3.0 as well.

[SparkQA]

Test build #122872 has finished for PR 28589 at commit b4d215e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Merged to master/3.0.

[HyukjinKwon]

Thank you @dongjoon-hyun.

[viirya]

Yea, LGTM too.

[tgravescs]

@HyukjinKwon ResourceInformation is now in a different module between spark 3.0 and 3.1. Even though users shouldn't be constructing one its still a public api. I left it the way it was to keep it consistent. It seems like we should put them in the same place between 3.0 and 3.1

[HyukjinKwon]

@tgravescs, yeah, I understand the initial intention. The problem I would like to address here is that don't expose ResourceInformation under pyspark with a minimised change as it became a separate package in the master - it will be difficult to hide it back once it's exposed.

To end users, there are no differences. We can even move this file into a separate package for consistency after the release. I thought it's not worthwhile to do it, and also given RC period. So I came up with a minimised fix.

If you strongly feel it should be as a package for dev purpose consistently in branch-3.0 too, I wouldn't mind.