[SPARK-39988][CORE] Use try-with-resource to ensure DBIterator is close after use
#37420
Conversation
LuciferYang
changed the title
try-with-resource to ensure DBIterator is close after use in RemoteBlockPushResolver, YarnShuffleService and ExternalShuffleBlockResolvertry-with-resource to ensure DBIterator is close after use in RemoteBlockPushResolver, YarnShuffleService and ExternalShuffleBlockResolver
LuciferYang
changed the title
try-with-resource to ensure DBIterator is close after use in RemoteBlockPushResolver, YarnShuffleService and ExternalShuffleBlockResolvertry-with-resource to ensure DBIterator is close after use in RemoteBlockPushResolver, YarnShuffleService and ExternalShuffleBlockResolver
| throws IOException { | ||
| ConcurrentMap<AppExecId, ExecutorShuffleInfo> registeredExecutors = Maps.newConcurrentMap(); | ||
| if (db != null) { | ||
| DBIterator itr = db.iterator(); |
There was a problem hiding this comment.
The methods reloadActiveAppAttemptsPathInfo and reloadFinalizedAppAttemptsShuffleMergeInfo should be Spark 3.4 only. Please let me know if this PR needs to be backport to other branch
|
cc @dongjoon-hyun @mridulm Changes of this pr is for bug fix, which is not within the refactor scope of SPARK-38909 |
LuciferYang
changed the title
try-with-resource to ensure DBIterator is close after use in RemoteBlockPushResolver, YarnShuffleService and ExternalShuffleBlockResolvertry-with-resource to ensure DBIterator is close after use
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Looks reasonable to me, but we had better ping more people in this area, @LuciferYang .
|
Merged to master. |
Sorry for not reply at the weekend, this issue exists in the historical version, do we need a backport? |
|
No, I don't think so for now, @LuciferYang . Let's keep this in master branch for Apache Spark 3.4.0 first. |
|
OK |
3 participants
What changes were proposed in this pull request?
This pr use
try-with-resourceto ensureDBIteratoris close after use inRemoteBlockPushResolver,YarnShuffleServiceandExternalShuffleBlockResolverto avoid resource leakage.Why are the changes needed?
Avoid
DBIteratorresource leakage.Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions