Skip to content

[SPARK-41193][SQL][TESTS] Ignore collect data with single partition larger than 2GB bytes array limit in DatasetLargeResultCollectingSuite #38704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 1 commit into from
Closed

[SPARK-41193][SQL][TESTS] Ignore collect data with single partition larger than 2GB bytes array limit in DatasetLargeResultCollectingSuite #38704

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2251,7 +2251,11 @@ class DatasetLargeResultCollectingSuite extends QueryTest
with SharedSparkSession {

override protected def sparkConf: SparkConf = super.sparkConf.set(MAX_RESULT_SIZE.key, "4g")
test("collect data with single partition larger than 2GB bytes array limit") {
// SPARK-41193: Ignore this suite because it cannot run successfully with Spark
// default Java Options, if user need do local test, please make the following changes:
// - Maven test: change `-Xmx4g` of `scalatest-maven-plugin` in `sql/core/pom.xml` to `-Xmx10g`
// - SBT test: change `-Xmx4g` of `Test / javaOptions` in `SparkBuild.scala` to `-Xmx10g`
ignore("collect data with single partition larger than 2GB bytes array limit") {
Copy link
Contributor

@mridulm mridulm Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liuzqt, I know this was iterated on multiple times to get it to work - instead of the shared local spark session, did it work locally when using a local spark cluster instead ?

Copy link
Contributor

@liuzqt liuzqt Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes @LuciferYang is right, need to change -Xmx4g to -Xmx10g to make it work (it works for both shared local session and local cluster, but without the change neither work).

Thanks for the fix! Previously I only tested this using IDE and I guess it increased the mem under the hood......Sorry for the inconvenience.

Copy link
Contributor Author

@LuciferYang LuciferYang Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So how do we move forward? This is a blocking for developers

Copy link
Contributor

@liuzqt liuzqt Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can leave it as ignore for now with the comments about using larger mem to make it work. I'm not sure if we're able to configure the build args for a specific test suite.

// This test requires large memory and leads to OOM in Github Action so we skip it. Developer
// should verify it in local build.
assume(!sys.env.contains("GITHUB_ACTIONS"))
Expand Down