Processing multiple files in nodejs in parallel

Question

I am working on a large application in node.js in which around 5000 users are required to upload excel files on a daily basis. Once a file is uploaded, app needs to evaluate that file and after some db validations, process these files....I am able to upload and process single file using multer but when multiple users are uploading, the system just went into a deadlock...I am using SQL server database. Is there any solution to handle this situation.

Thanks in advance.

Answer 1

You need to process these incoming files one at a time (or two or some controlled number). And you don't want to do that processing from inside the same nodejs/javascript instance that's serving your web app. Things will bog down even if you can prevent deadlocks.

Here's how I've handled this problem successfully.

Get the web server (multer) to write the incoming uploads to files in a particular directory somewhere. Name them with names that identify them. For example your names could take this form

/var/myapp/incoming/2020-10-01T19-32-20U98765

This is a file uploaded at a certain time by user 98765.

Write a standalone program to look at the incoming directory, pick a file from it, process the file, and move it to a processed directory.