Send and receive a large json object in http

Question

I have a configuration is such a way that.

Client application server running on Nodejs with expressjs. API server is running on Nodejs with expressJs and I am using postgresql database.

When my client has requested data from api server, api server is throwing an error

"*Error is  RangeError: Invalid string length*". 

I guess the reason for it is, api is tying to respond to client with an extremely large array of Json objects, which it has fetched from database.

Now my question is, How can I send such an extremely large Json object in http response? And if sent successfully, how can I handle it at the client side?

I came across JSONStream module. But, I couldn't find a good example which I can pick a implement in my scenario.

Answer 1

Whenever we have to deal with larger data and we are facing issues due to it, in such scenarios data should always be fetched in chunks, rather than loading all data at once. So in your scenario, I suggest you to load data in parts.

Answer 2

I think BodyParser will help..

var bodyParser = require('body-parser');

 app.use(bodyParser.urlencoded({
    parameterLimit: 100000,
    limit: '50mb',
    extended: true
  }));

This will increase the size of json over http requests and response

You can also check:-

const fs = require('fs');
const server = require('http').createServer();

server.on('request', (req, res) => {
  const src = fs.createReadStream('./big.file');
  src.pipe(res);
});

server.listen(8000);

Ref- Freecodecamp link