ActiveRecord transactions: record created but not found in database

Question

We're using Rails v6.0.2.1 on Jruby with managed PostgreSQL (using JDBC adapter) hosted on DigitalOcean. Lately, we have been seeing issues where a certain transaction created some record, threw the Id to Sidekiq for further processing but the record isn't there in the database and Sidekiq fails since the record is not found in db.

There are some Concurrent::Future calls as well inside the transaction. DigitalOcean doesn't show any deadlocks in the said time period. And currently, DO doesn't provide a dump of PostgreSQL logs. How do we debug this?

While looking at Rails production logs, we found out that those transactions didn't log any BEGIN...COMMIT OR ROLLBACK message either.

Answer 1

you might have been using after_save callback on your models I assume? anyway, the issue is that Sidekiq is fast. Sometimes, too fast.

What is happening is that Sidekiq is picking up the job before the transaction is committed. That will trigger the error you are seeing (e.g. RecordNotFound).

You have two solutions: simply retry the job, on a second pass, the record would most likely be in the database again or you can go move to after_commit callback and these errors will disappear.

This issue was discussed here in the past