I am currently working with throught2 module to write gulp plugin to replace something in the buffer, below is a simplified version of it
module.exports = function(replaceData: DataChange | DataChange[]) {
let count = 0;
return through.obj(function (file: any, encoding: any, callback: any) {
this.push(file);
callback(null, file);
console.log('@COUNT: ', ++count);
});
};
I found out that it will pick up only 8 files and will continue, ignoring the others, my output will be like:
@COUNT: 1
@COUNT: 2
...
@COUNT: 8
If I don't return anything in callback
module.exports = function(replaceData: DataChange | DataChange[]) {
let count = 0;
return through.obj(function (file: any, encoding: any, callback: any) {
this.push(file);
callback();
console.log('@COUNT: ', ++count);
});
};
it will process 16 files. How to increase the number of files through will pick up?
What I found out for through2 is expecting streams that inherits from Readable & Writable.
Default Readable/Writable hihghWaterMark is 16K. Without a consumer for the through2 transform stream relieving pressure, it hits the high water mark and buffers after 16 records. In case of returning file in callback(case 1) it is 8 files in and 8 files out, that's why it's picking only 8.
The solution will be to pass to through2 options, i.e.
and it will pick up 32 files in my case.
Supported options include:
Hope this will help you and save your time