Skip to content

[SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo #2792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liyezhang556520 wants to merge 2 commits into from
Closed

[SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo #2792

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0380a32
[SPARK-3941][CORE] _remainingmem should not increase twice when updat…
liyezhang556520 Oct 14, 2014
3d487cc
make the code concise
liyezhang556520 Oct 17, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 14 additions & 11 deletions core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,18 +455,21 @@ private[spark] class BlockManagerInfo(

updateLastSeenMs()

if (_blocks.containsKey(blockId)) {
// The block exists on the slave already.
val originalLevel: StorageLevel = _blocks.get(blockId).storageLevel

if (originalLevel.useMemory) {
_remainingMem += memSize
}
}

if (storageLevel.isValid) {
/* isValid means it is either stored in-memory, on-disk or on-Tachyon.
* But the memSize here indicates the data size in or dropped from memory,
// isValid means it is either stored in-memory, on-disk or on-Tachyon.

if (_blocks.containsKey(blockId)) {
// The block exists on the slave already.
val blockStatus: BlockStatus = _blocks.get(blockId)
val originalLevel: StorageLevel = blockStatus.storageLevel
val originalMemSize: Long = blockStatus.memSize

@andrewor14 andrewor14 Oct 17, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old code seems a little more concise


if (originalLevel.useMemory) {
_remainingMem += originalMemSize
}
}

/* The memSize here indicates the data size in or dropped from memory,
* tachyonSize here indicates the data size in or dropped from Tachyon,
* and the diskSize here indicates the data size in or dropped to disk.
* They can be both larger than 0, when a block is dropped from memory to disk.
Expand Down