Skip to content

[SPARK-19839][Core]release longArray in BytesToBytesMap #17180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhzhan wants to merge 2 commits into from
Closed

[SPARK-19839][Core]release longArray in BytesToBytesMap #17180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d315c60
release longArray in BytesToBytesMap
Mar 6, 2017
2403c30
fix unit test
Jul 27, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,11 @@ private MapIterator(int numRecords, Location loc, boolean destructive) {
this.destructive = destructive;
if (destructive) {
destructiveIterator = this;
// longArray will not be used anymore if destructive is true, release it now.
if (longArray != null) {
freeArray(longArray);
longArray = null;
}
}
}

Expand Down
5 changes: 3 additions & 2 deletions ...e/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,9 +127,10 @@ class UnsafeFixedWidthAggregationMapSuite
PAGE_SIZE_BYTES
)
val groupKey = InternalRow(UTF8String.fromString("cats"))
val row = map.getAggregationBuffer(groupKey)

// Looking up a key stores a zero-entry in the map (like Python Counters or DefaultDicts)
assert(map.getAggregationBuffer(groupKey) != null)
assert(row != null)
val iter = map.iterator()
assert(iter.next())
iter.getKey.getString(0) should be ("cats")
Expand All @@ -138,7 +139,7 @@ class UnsafeFixedWidthAggregationMapSuite

// Modifications to rows retrieved from the map should update the values in the map
iter.getValue.setInt(0, 42)
map.getAggregationBuffer(groupKey).getInt(0) should be (42)
row.getInt(0) should be (42)

map.free()
}
Expand Down