NodeJS Writable streams writev alternating between 1 and highWaterMark chunks

Question

So I have a stream that generates data and one that writes them to the database. Writing to the databse is slow. I use the writev function to write a batch of 3000 chunk at once.

const generator = new DataGenerator(); // extends Readable
const dbWriter = new DBWriter({ highWaterMark: 3000 }); // extends Writable, implements _writev method

pipeline(
  generator,
  dbWriter 
)

But when I log chunk counts in the _writev method, I get following output:

1
2031
969
1
1635
1365
1
1728
1272
1
...

I understand the first line is 1. A chunk comes, DB starts writing. 2031 chunks come in the meantime.

Then DB starts writing the 2031 chunks and another 969 chunks come in the meantime, not 3000. And then in the next step, only 1 is written again. Like if receiving chunks to buffer would reset only when everything is written, not when the 3000 buffer is not full.

What I would expect:

1
2031
3000
3000
3000
...
3000
123

Why?

Answer 1

Well because there is no warranty that you will get 3000 chunk of data, it does tell the limit of the inner buffer that your writable stream has. It is okay that you can receive an arbitrary amount of data because read stream knows nothing about your buffer size. Best regards.