How to Bulk import with validations and create versions using paper_trail?

Question

I am having an array of hashes(around 20k) that I need to import but with validations and have to create the versions as well using paper_trail.

arr = [{some_1_id: '1', some_2_id: '2', some_3_id: '3', amount: '123'}, {some_1_id: '1', some_2_id: '2', some_3_id: '3', amount: '123'}]

I have an array of hashes like this. In this, we can expect around 20k elements that need to be saved to DB.

I have used activerecord-import but it's bypassing all of the callbacks including paper_trail callbacks.

I tried to run the callbacks like this

books.each do |book|
  book.run_callbacks(:save) { false }
  book.run_callbacks(:create) { false }
end
Book.import(books)

But it's not saving the paper_trail versions correctly.

What I did is that

arr.each do |a|
 valid_books = []
 book_obj = Book.new
 # validate some_1_id
 # validate some_2_id   
 # validate some_3_id
 # Some other validations
 if book_obj.valid?
   valid_books << book_obj
 end
end
 # Run call backs 
 Book.import valid_books

It's working as expected but performance is not good at all. its taking more than 30 secs for 10k records.

Thanks in advance.

Answer 1

its taking more than 30 secs for 10k records.

30s is within the expected performance range for validating and creating 10k versionsed records normally (ie. new, valid?, save with PT callbacks). In some applications, 30s would be considered fast, especially when validations and other callbacks perform queries of their own.

I recommend putting the import process in a background job and simply asking the user to wait.

If the import is already in a background job, or a bg. job is not possible, you could explore using activerecord-import twice: first to create 30k Book records, and then to create 30k PaperTrail::Version records. This is an advanced technique, using private (unsupported) API of PT. To correctly instantiate the PaperTrail::Version objects, use PaperTrail::Events::Create, as PaperTrail::RecordTrail#build_version_on_create does. If you are successful with this technique, I'd gladly review a PR adding it to our README.