Pg-promise stops opening new connections, no error message

581 Views Asked by At

In my express-js app pg-promise sometimes stops accepting connections, even for simple things as a select now() query. It's very random. I can run a load test and have 100 concurrent requests run for 10 seconds with response times of 100+ req/s. Then suddenly it just stops working and a single curl request won't work even though it does nothing other than making a select now() query.

When this happens it sometimes it works by restarting the node app. Sometimes even that doesn't help.

I can always access to express app for routes not accessing the database, so I'm fairly certain this has to do with the database connection.

The connection options to pgp looks like this (details removed):

{
    host: '111.111.111.111',
    port: 5432,
    database: 'database-name',
    user: 'database-user',
    password: 'super secret password'
}

My simple express route that test the database looks like this:

app.get(apiPrefix + '/fail', async (req: express.Request, res: express.Response, next: NextFunction) => {
    await db.task(uuid(), async (tx) => {
        const time = await tx.one('select now() as time');

        res.json({status: 'OK', time: time.time});
    });
});

I also have a middleware that handles errors and logs them, but nothing is logged when this happens. Sometimes the task event on pgpFunc.IInitOptions fires, but never with the "finish" prop set to true. The error event never fires.

It feels like something is blocking the connection pool, but I can't find out what and why.

Example output from pg-monitor when it fails:

14:26:43 connect(***@***); useCount: 0
14:26:43 task(Auth: acb606e2-a79f-42a4-afe7-f94c8e35d2e8)/start
14:26:43 connect(***@***); useCount: 0
14:26:43 task(Auth: eb28be63-dfda-43a2-9884-05415aa54fbe)/start
14:26:43 connect(***@***); useCount: 0
14:26:43 task(Auth: 59ec38c4-be8f-498b-a77b-3a96ecddf41d)/start
14:26:43 connect(***@***); useCount: 0
14:26:43 task(Auth: f024626a-5e59-4675-a3f2-825ed78dd35d)/start
14:26:43 connect(***@***); useCount: 0
14:26:43 task(Auth: df456fad-ed80-4110-995f-0260db4ec118)/start
14:26:43 connect(***@***); useCount: 0
0

There are 0 best solutions below