S3 gateway memory skyrocketing issue

[HavAnIdea]

Discussed in #4502

^{Originally posted by HavAnIdea March 14, 2024}
I encountered a memory skyrocketing issue when I initiated the S3 gateway for reading and writing through Spark.
However, when I use DataX or JuiceFS sync, there are no issues encountered.

How should I troubleshoot the issue?

usually

reading and writing through Spark

[HavAnIdea]

gateway log

[SandyXSD]

I also met a similar memory issue when using gateway and have fixed it by #4507. Could you try it?

[HavAnIdea]

I also met a similar memory issue when using gateway and have fixed it by #4507. Could you try it?

I see that your commit has been merged into the main branch. Should I compile and use it locally? I'm not sure if setting flags would be helpful for reading and writing with Spark, but I can give it a try.

[davies]

Can you grab the heap profile for analysis?