I'm trying to add workers to my Node.JS application so that I can run jobs after an endpoint gets hit and eventually process files.
I am trying to use the npm package workerpool but when I call my workerpool to execute a function I get the following error:
Error running analysis: Error: Worker is terminated
I setup my workerpool in worker.ts
import workerpool from "workerpool"
export const setupWorkers = async () => {
try {
const filepath = __dirname + "/analysisWorker.ts"
const pool = workerpool.pool(filepath, {
minWorkers: 2,
maxWorkers: 4,
maxQueueSize: 20,
workerType: "thread",
emitStdStreams: true,
})
console.log(`Total workers: ${pool.stats().totalWorkers}`)
return pool
} catch (err) {
console.error("Error when setting up workers")
}
}
the filepath points to my analysisWorker.ts file which only has this in it:
import workerpool from "workerpool"
function runAnalysis() {
console.log("job started")
return 1000
}
// NOTE: I have tried removing this aswel
workerpool.worker({
runAnalysis: runAnalysis,
})
I then setup my workers in my app.ts file like so:
...
app.use(cookieParser("test_123"))
app.use(express.json())
app.use(cors())
const startApp = async () => {
// Setup workers
const workerpool = await setupWorkers()
// Attach workerpool to app
app.set("workerpool", workerpool)
// Connect to database
await connect()
// Setup routes
routes(app)
// Error handling
app.use(errorHandler)
app.listen(PORT, async () => {
console.log(
`Server started on port ${PORT} in ${process.env.NODE_ENV} mode`
)
})
}
startApp()
When I start the app I get the expected console log of "Total workers: 2" in my terminal.
Note I am also using app.set
to set the workerpool so I can use it in the rest of my project.
Finally, in my controller file productController.ts, I have a product class with the following function which is ran when I hit localhost:3000/products
with a POST.
public async create(req: Request, res: Response, next: NextFunction) {
const product = await productService.addProduct(req.body.name)
const workerpool = req.app.get("workerpool")
workerpool
.exec("runAnalysis")
.then((result: number) => {
console.log("result: ", result)
})
.catch((err: Error) => {
console.error("Error running analysis: ", err)
})
console.log(workerpool.stats().totalWorkers)
res.send({ status: "OK", data: product, count: 1 })
}
Notice the catch
block is what is producing the error above, however the console.log
directly after it prints 2 to the terminal, which means I still have active workers.
Can anyone see what I am doing wrong, I have tried looking at the docs and can't figure out why my workers are terminated but still active.