Increase SPARK_WORKER_MEMORY to 1g

[adrianyorke]

Introduction

Hi there, thanks for helping the project! We are doing our best to help the community to learn and practice
parallel computing in distributed environments through our projects. ✨

Feature

Please fill the template below.

Description

Running this in 2024 on both Windows and Linux platforms, I had the same issue. 512mb doesn't seem to be enough for the workers. I simply doubled the memory to 1g and the errors went away in both Windows and Linux. I will create a PR for this simple configuration change.

Comments (optional)

My Windows version is 11.
My Linux is the latest Linux Mint 22 'Wilma'.