[SPARK-30480][PYSPARK][TESTS] Fix 'test_memory_limit' on pyspark test

[HeartSaVioR]

What changes were proposed in this pull request?

This patch increases the memory limit in the test 'test_memory_limit' from 1m to 8m.
Credit to @srowen and @HyukjinKwon to provide the idea of suspicion and guide how to fix.

Why are the changes needed?

We observed consistent Pyspark test failures on multiple PRs (#26955, #26201, #27064) which block the PR builds whenever the test is included.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Jenkins builds passed in WIP PR (#27159)

[HeartSaVioR]

In case if the PR build doesn't take the test (I think it shouldn't, but), I'll also trigger the build in #27159 couple more times.

[viirya]

any idea on why this change is needed now?

[SparkQA]

Test build #116449 has finished for PR 27162 at commit 7c77ec7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HeartSaVioR]

Sorry I have no idea. I guess there's some change on Jenkins env. or code change might affect, but it's not easy to figure out why it's failing in either way only to investigate on log file. The change also doesn't hurt.

cc. @shaneknapp Do you have any idea on changes in Jenkins env.? The test suddenly started to fail yesterday, and we seem to have to increase memory limit on the test to fix it.

[HyukjinKwon]

I am going to merge to unblock those PRs.

Merged to master.

[HeartSaVioR]

Thanks for the quick reaction! Let's see builds for these PRs pass.

[HeartSaVioR]

Looks like the builds are still failing from previous PRs, though I'm not sure why the builds in new WIP PR were succeeded.

[HyukjinKwon]

I'll revert this and take a look tomorrow if that's still persistent.

[HyukjinKwon]

Reverted.

[shaneknapp]

Sorry I have no idea. I guess there's some change on Jenkins env. or code change might affect, but it's not easy to figure out why it's failing in either way only to investigate on log file. The change also doesn't hurt.

cc. @shaneknapp Do you have any idea on changes in Jenkins env.? The test suddenly started to fail yesterday, and we seem to have to increase memory limit on the test to fix it.

nope, the jenkins python envs haven't changed at all and i'm not sure what's going on here.

i'll take a closer look monday...