sf data query --bulk throws Fatal Error Javascript Heap out of memory

[AllanOricil]

Is your feature request related to a problem? Please describe.

Bulk query stores all records in memory, instead of writing them to disk

src: https://github.com/jsforce/jsforce/blob/95932f288a59a0b68ff9fe8cc1a94523a4d0fb85/src/api/bulk.ts#L1077-L1099

src: https://github.com/jsforce/jsforce/blob/95932f288a59a0b68ff9fe8cc1a94523a4d0fb85/src/api/bulk.ts#L1244-L1264

this could throw an exception like Process out of Memory Exception if someone tries to export a few million records, because all records are stored in memory inside of a record[] array. For instance,

sf data query --query "SELECT Id FROM Contact" --bulk

src: https://github.com/salesforcecli/plugin-data/blob/b56e778f6c9ef8affbc0719cafb629b07d19ce2c/src/commands/data/query.ts#L135-L153

This type of exception can't be captured by a try/catch. The node runtime would exit 1 straight away.

This issue does not happen when using jsforce bulk v1 implementation because it was developed with streams in mind.

src: https://github.com/jsforce/jsforce/blob/95932f288a59a0b68ff9fe8cc1a94523a4d0fb85/src/api/bulk.ts#L959-L983

What are you trying to do
Export a few million records using sfdx force:data:query --bulk

Describe the solution you'd like

1 . jsforce must accept a write stream so that memory could be released after its data is written to disk. Additionally, memory can be released when this.records[] reaches N kbs of data.
2. sfdx should create a write stream when the -b flag is true.

Describe alternatives you've considered
N/a

Additional context
N/a

[github-actions]

Thank you for filing this feature request. We appreciate your feedback and will review the feature at our next grooming or sprint planning session. We prioritize feature requests with more upvotes and comments.

[git2gus]

This issue has been linked to a new work item: W-12677563

[AllanOricil]

@mshanemc @RodEsp
I think you could take a look at this since you are maintaining jsforce 2.0

[RodEsp]

Heya @AllanOricil, I'm not involved with JSForce anymore but I'm sure Shane can take a look.

[AllanOricil]

Experiment with 103k Product2 records

This is the result when using the default heap size

This is the result after configuring the heap size to 128Mb

[iowillhoit]

Thanks for this also @AllanOricil, I'll add this to a JSForce category in our back log

[AllanOricil]

These PRs fix this bug

[cristiand391]

closing this, for big queries we added a new data export bulk commands:
https://github.com/forcedotcom/cli/tree/main/releasenotes/#2626-october-16-2024

bulk mode for data query will be deprecated in a future release.