Skip to content

[SPARK-43231][ML][PYTHON][CONNECT][TESTS] Reduce the memory requirement in torch-related tests #40874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhengruifeng wants to merge 11 commits into from
Closed

[SPARK-43231][ML][PYTHON][CONNECT][TESTS] Reduce the memory requirement in torch-related tests #40874

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0aa206d
init
zhengruifeng Apr 20, 2023
0ecfb2a
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 21, 2023
20cd526
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 21, 2023
ac0dc78
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 21, 2023
bdbfaa6
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 22, 2023
a69dd0c
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 24, 2023
4fc3f94
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 24, 2023
9b33d36
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 24, 2023
1af98d4
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 24, 2023
f726863
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 25, 2023
9bcc7da
Merge branch 'apache:master' into torch_reduce_memory
zhengruifeng Apr 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions python/pyspark/ml/tests/connect/test_parity_torch_distributor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ def setUpClass(cls):
builder = builder.config(
"spark.driver.resource.gpu.discoveryScript", cls.gpu_discovery_script_file_name
)
cls.spark = builder.remote("local-cluster[2,2,1024]").getOrCreate()
cls.spark = builder.remote("local-cluster[2,2,512]").getOrCreate()

@classmethod
def tearDownClass(cls):
Expand Down Expand Up @@ -126,7 +126,7 @@ def setUpClass(cls):
builder = builder.config(
"spark.worker.resource.gpu.discoveryScript", cls.gpu_discovery_script_file_name
)
cls.spark = builder.remote("local-cluster[2,2,1024]").getOrCreate()
cls.spark = builder.remote("local-cluster[2,2,512]").getOrCreate()

@classmethod
def tearDownClass(cls):
Expand Down
8 changes: 6 additions & 2 deletions python/pyspark/ml/torch/tests/test_distributor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,8 @@ def get_local_mode_conf():
return {
"spark.test.home": SPARK_HOME,
"spark.driver.resource.gpu.amount": "3",
"spark.driver.memory": "512M",
"spark.executor.memory": "512M",
}


Expand All @@ -158,6 +160,8 @@ def get_distributed_mode_conf():
"spark.task.cpus": "2",
"spark.task.resource.gpu.amount": "1",
"spark.executor.resource.gpu.amount": "1",
"spark.driver.memory": "512M",
"spark.executor.memory": "512M",
}


Expand Down Expand Up @@ -412,7 +416,7 @@ def setUpClass(cls):
"spark.driver.resource.gpu.discoveryScript", cls.gpu_discovery_script_file_name
)

sc = SparkContext("local-cluster[2,2,1024]", cls.__name__, conf=conf)
sc = SparkContext("local-cluster[2,2,512]", cls.__name__, conf=conf)
cls.spark = SparkSession(sc)

@classmethod
Expand Down Expand Up @@ -502,7 +506,7 @@ def setUpClass(cls):
"spark.worker.resource.gpu.discoveryScript", cls.gpu_discovery_script_file_name
)

sc = SparkContext("local-cluster[2,2,1024]", cls.__name__, conf=conf)
sc = SparkContext("local-cluster[2,2,512]", cls.__name__, conf=conf)
cls.spark = SparkSession(sc)

@classmethod
Expand Down