feat: Adding threshold for logging memory pool allocations #14437

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

kletkavrubashku wants to merge 1 commit into facebookincubator:main from kletkavrubashku:export-D79902261

Contributor

kletkavrubashku commented Aug 12, 2025 •

edited

Loading

Summary:

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Differential Revision: D79902261

Test plan:

SET session native_debug_memory_pool_name_regex="^op\.11\.\d+\.\d+\.TableScan$"
SET session native_debug_memory_pool_warn_threshold_bytes="1MB"

Example logs:

W0810 21:42:07.801401  1256 [Driver165] MemoryPool.cpp:1230] [MEM] [MemoryPool] Memory pool 'op.11.0.13.TableScan' exceeded warning threshold of 1048576 bytes with allocation of 16384 bytes, resulting in total size of 1054966 bytes.
--
======== Allocation Stack ========
# 0  facebook::velox::memory::MemoryPoolImpl::recordAllocDbg(void const*, unsigned long)
# 1  facebook::velox::memory::MemoryPoolImpl::allocate(long, std::optional<unsigned int>)
# 2  boost::intrusive_ptr<facebook::velox::Buffer> facebook::velox::AlignedBuffer::allocate<int>(unsigned long, facebook::velox::memory::MemoryPool*, std::optional<int> const&, bool)
# 3  void facebook::velox::dwio::common::SelectiveColumnReader::ensureValuesCapacity<int>(int, bool)
# 4  void facebook::velox::dwio::common::SelectiveColumnReader::prepareRead<int>(long, folly::Range<int const*> const&, unsigned long const*)
# 5  facebook::velox::dwrf::SelectiveStringDictionaryColumnReader::read(long, folly::Range<int const*> const&, unsigned long const*)
...
# 22 start_thread
# 23 __clone3

======= Current Allocations ======
Found 76 allocations:
======== 1 allocations of 256.00KB total size ========
# 0  facebook::velox::memory::MemoryPoolImpl::recordAllocDbg(void const*, unsigned long)
# 1  facebook::velox::memory::MemoryPoolImpl::allocateZeroFilled(long, long)
# 2  facebook::velox::dwio::common::compression::PagedInputStream::prepareOutputBuffer(unsigned long)
# 3  facebook::velox::dwio::common::compression::PagedInputStream::readOrSkip(void const**, int*)
...
# 26 execute_native_thread_routine
# 27 start_thread
# 28 __clone3

======== 1 allocations of 127.96KB total size ========
# 0  facebook::velox::memory::MemoryPoolImpl::recordAllocDbg(void const*, unsigned long)
# 1  facebook::velox::memory::MemoryPoolImpl::allocateZeroFilled(long, long)
...

netlify bot commented Aug 12, 2025 •

edited

Loading

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`9d7a58b`
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/68a90f21fb6c040008585ca6

meta-cla bot added the CLA Signed label

Contributor

facebook-github-bot commented Aug 12, 2025

This pull request was exported from Phabricator. Differential Revision: D79902261

facebook-github-bot added the fb-exported label

kletkavrubashku marked this pull request as draft

August 12, 2025 05:50

kletkavrubashku changed the title ~~Adding threshold for logging memory pool allocations~~ feat: Adding threshold for logging memory pool allocations

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [velox][memory] Adding threshold for logging memory pool allocations"

2b8dafc

Summary:
I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Differential Revision: D80066283

kletkavrubashku mentioned this pull request

feat: Add session property for debugMemoryPoolWarnThresholdBytes prestodb/presto#25750

Merged

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [velox][memory] Adding threshold for logging memory pool allocations" (…

87e7730

…prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Differential Revision: D80066283

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

753692a

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Differential Revision: D80066283

kletkavrubashku force-pushed the export-D79902261 branch from a870bcc to 02ab628 Compare

August 13, 2025 00:34

kletkavrubashku added a commit to kletkavrubashku/velox that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

02ab628

…or#14437)

Summary:

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Differential Revision: D79902261

Contributor

facebook-github-bot commented Aug 13, 2025

This pull request was exported from Phabricator. Differential Revision: D79902261

kletkavrubashku added a commit to kletkavrubashku/velox that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

1ffd416

…or#14437)

Summary:

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Differential Revision: D79902261

kletkavrubashku force-pushed the export-D79902261 branch from 02ab628 to 1ffd416 Compare

August 13, 2025 00:46

Contributor

facebook-github-bot commented Aug 13, 2025

This pull request was exported from Phabricator. Differential Revision: D79902261

kletkavrubashku added a commit to kletkavrubashku/velox that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

8c721de

…or#14437)

Summary:

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Differential Revision: D79902261

kletkavrubashku force-pushed the export-D79902261 branch from 1ffd416 to 8c721de Compare

August 13, 2025 01:38

Contributor

facebook-github-bot commented Aug 13, 2025

This pull request was exported from Phabricator. Differential Revision: D79902261

kletkavrubashku force-pushed the export-D79902261 branch from 8c721de to 855f5ac Compare

August 13, 2025 19:11

kletkavrubashku added a commit to kletkavrubashku/velox that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

855f5ac

…or#14437)

Summary:

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Differential Revision: D79902261

Contributor

facebook-github-bot commented Aug 13, 2025

This pull request was exported from Phabricator. Differential Revision: D79902261

kletkavrubashku force-pushed the export-D79902261 branch from 855f5ac to c5937da Compare

August 14, 2025 18:52

kletkavrubashku added a commit to kletkavrubashku/velox that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

c5937da

…or#14437)

Summary:

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Differential Revision: D79902261

Contributor

facebook-github-bot commented Aug 14, 2025

This pull request was exported from Phabricator. Differential Revision: D79902261

kletkavrubashku added a commit to kletkavrubashku/velox that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

7dbced0

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Differential Revision: D79902261

kletkavrubashku force-pushed the export-D79902261 branch from c5937da to 7dbced0 Compare

August 14, 2025 18:55

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

e542f04

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Differential Revision: D80066283

kletkavrubashku marked this pull request as ready for review

August 18, 2025 18:56

kletkavrubashku force-pushed the export-D79902261 branch from 7dbced0 to 1819d1c Compare

August 18, 2025 18:59

Contributor

facebook-github-bot commented Aug 23, 2025

This pull request was exported from Phabricator. Differential Revision: D79902261

facebook-github-bot closed this in

255da1a

facebook-github-bot added the Merged label

Contributor

facebook-github-bot commented Aug 23, 2025

This pull request has been merged in 255da1a.

kletkavrubashku deleted the export-D79902261 branch

August 23, 2025 21:04

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

8378a0f

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Differential Revision: D80066283

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

7ca018b

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Differential Revision: D80066283

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

156acb2

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Reviewed By: tanjialiang

Differential Revision: D80066283

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

e692583

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Reviewed By: tanjialiang

Differential Revision: D80066283

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

e0d529f

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Reviewed By: tanjialiang

Differential Revision: D80066283

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

08c2d09

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Reviewed By: tanjialiang

Differential Revision: D80066283

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

c744e7d

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Reviewed By: tanjialiang

Differential Revision: D80066283

kletkavrubashku added a commit to kletkavrubashku/presto that referenced this pull request


          [presto][native] Add session property for debug_memory_pool_warn_thre…

c91b80e

…shold_bytes (prestodb#25750)

Summary:

I added threshold for logging memory pool allocations": facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure the threshold.

Reviewed By: tanjialiang

Differential Revision: D80066283

kletkavrubashku added a commit to prestodb/presto that referenced this pull request


          feat: Add session property for debugMemoryPoolWarnThresholdBytes (#25750

ac1f4f7

)

Summary:
I added threshold for logging memory pool allocations":
facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure
the threshold.

Differential Revision: D80066283

dan13bauer pushed a commit to dan13bauer/velox that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

d8e04bb

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

7a4c9d2

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

wypb pushed a commit to wypb/velox that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

c293e3a

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

0a37ca8

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

2ab718c

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

a91d4d7

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

40371e4

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

143ed2a

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

a581d79

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

148d759

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

2284dae

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

da4ee8c

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/velox-exchange that referenced this pull request


          Adding threshold for logging memory pool allocations (facebookincubat…

107d80b

…or#14437)

Summary:
Pull Request resolved: facebookincubator#14437

We have a leak check in memory pool the prints all current allocations along with call sites.
In this diff I'm expanding that functionality and adding possibility to log that information once we reach configurable threshold. It's supposed to be used for debugging, when we want to know how exactly the memory has been used in particular memory pool. Adding this only for leaf nodes for now.

Reviewed By: tanjialiang

Differential Revision: D79902261

fbshipit-source-id: 98dfc2ed25e45fa3a7881fdaf71743677d012173

lga-zurich pushed a commit to lga-zurich/presto-exchange that referenced this pull request


          feat: Add session property for debugMemoryPoolWarnThresholdBytes (pre…

401d1de

…stodb#25750)

Summary:
I added threshold for logging memory pool allocations":
facebookincubator/velox#14437
In this adding I'm adding corresponding session property to configure
the threshold.

Differential Revision: D80066283

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed fb-exported Merged