Ability to set highWaterMark on transform stream?

[olalonde]

• Version: v6.6.0
• Platform: OS X El Capitan
• Subsystem: Stream

I noticed that transform streams buffer up to 16kb (the default high watermark for read/write streams) even when they are not piping anywhere. I looked through the code/docs and it seems this value can't be set by implementers. As I'm writing a library which relies on this behaviour (transform buffering even when not piping to anywhere), I'd like to make sure it's not going to change and also was wondering if it would be possible for the Transform constructor to accept the highWaterMark option as well.

[mscdex]

All of the constructors can already accept options, including highWaterMark. For example:

var Transform = require('stream').Transform;
var inherits = require('util').inherits;

function MyTransform(opts) {
  if (typeof opts !== 'object' || opts === null)
    opts = {};
  Transform.call(this, { highWaterMark: opts.highWaterMark });
  // Or just: `Transform.call(this, opts);`

  // Other initialization ...
}
inherits(MyTransform, Transform);

MyTransform.prototype._transform = function(chunk, enc, cb) {
  // ...
};

[olalonde]

Ok, thanks. Maybe I'm misunderstanding something then. I just tested the following code, which outputs 16384 instead of the expected 100 (highWaterMark) to the console:

var Transform = require('stream').Transform
var RandomStream = require('common-streams').RandomStream

class CustomTransform extends Transform {
  constructor() {
    super({
      highWaterMark: 100,
    })
  }

  _transform(chunk, enc, cb) {
    console.log(chunk.length)
    cb(null, chunk)
  }
}

const rand = new RandomStream(10 * 1024 * 1024)

rand.pipe(new CustomTransform())

Is highWaterMark a minimum which can possibly be exceeded?

[mscdex]

highWaterMark is just an indicator to communicate to upstream that no more data should be written/pushed. Obviously you can write more data and it will just continue to be buffered in memory. highWaterMark does not dictate the size of chunks passed to _transform()/_write().

[olalonde]

Ah ok, so it's more of a signal to upstream but not enforced, thanks.

[ghost]

You should set it on the read stream you want to connect to your transform stream, otherwise it is not applied. I got sometimes bigger chunks even when I set it, so you cannot rely on this. I expect in my code that the chunk size is totally random and wait for more chunks if it is too small.

[kennylbj]

How can I set the chunk size in _transform function?

[bbrzoska]

If you're here because your @aws-sdk/lib-storage Upload code started failing after upgrading node from v18 to v22, and have this line:

Readable.toWeb(stream)

then try changing it to:

Readable.from(stream)

Which fixes the problem for me.