[SPARK-32303][PYTHON][TESTS] Remove leftover from editable mode installation in PIP test

[HyukjinKwon]

What changes were proposed in this pull request?

Currently the Jenkins PIP packaging test fails as below intermediately:

Installing dist into virtual env
Processing ./python/dist/pyspark-3.1.0.dev0.tar.gz
Collecting py4j==0.10.9 (from pyspark==3.1.0.dev0)
  Downloading https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl (198kB)
Installing collected packages: py4j, pyspark
  Found existing installation: py4j 0.10.9
    Uninstalling py4j-0.10.9:
      Successfully uninstalled py4j-0.10.9
  Found existing installation: pyspark 3.1.0.dev0
Exception:
Traceback (most recent call last):
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 50, in install_given_reqs
    auto_confirm=True
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 816, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 505, in from_dist
    '(at %s)' % (link_pointer, dist.project_name, dist.location)
AssertionError: Egg-link /home/jenkins/workspace/SparkPullRequestBuilder@3/python does not match installed 

• [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame #29099 (comment) (amp-jenkins-worker-04)
• [SPARK-32293] Fix inconsistency between Spark memory configs and JVM option #29090 (comment) (amp-jenkins-worker-03)

Seems like the previous installation of editable mode affects other PRs.

This PR simply works around by removing the symbolic link from the previous editable installation. This is a common workaround up to my knowledge.

Why are the changes needed?

To recover the Jenkins build.

Does this PR introduce any user-facing change?

No, dev-only.

How was this patch tested?

Jenkins build will test it out.

[SparkQA]

Test build #125823 has finished for PR 29102 at commit fc648af.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[attilapiros]

retest this please

[SparkQA]

Test build #125825 has finished for PR 29102 at commit 49a827f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #125829 has finished for PR 29102 at commit 5879260.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[attilapiros]

retest this please

[SparkQA]

Test build #125837 has finished for PR 29102 at commit 5879260.

• This patch fails PySpark pip packaging tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

nit. left over -> a leftover?

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

I re-trigger Jenkins because this PR should be tested on Jenkins.

[SparkQA]

Test build #125855 has finished for PR 29102 at commit 5879260.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. I agree that the analysis and the solution. Also, as mentioned in the PR, this is common like the following.

• pypa/pip#3784

Merged to master.

[HyukjinKwon]

Thanks @dongjoon-hyun. I will keep monitoring this.