I am currently developing as system from using storm. Data will come in from a kafka spout to a few bolt the the exit will be to a botl which sets a tcp connection to a NodeJs server application.
There is no problem when testing with low input of data. The problem arises when I test it with a large amount of data which the stream seems to be unstable.
I tried two ways to set the connection: Setting the connection only once and setting the connection and close once done. When I set the connection once the stream is coming in so fast into my NodeJs application the data got messed up and hit a data format error.
Opening and closing a connection makes the stream into NodeJs really unstable, it is ok the first few kb but after a while it stop the stream the flood stop and flood thr stream.
I'm just wondering: is there is any other way other then setting a tcp connection from storm to NodeJs or is there a way to make a connection once to NodeJs and make the message que. I'm running out of ideas any help on this issue?