I'm retrieving a large list of attendees for an event from a Firebase database - roughly 4,000 attendees. Each is represented as a JSON object of roughly 500 bytes.
After retrieving the dataset, I am trying to store it in a persistent local store. Initially I tried using Lawnchair.js using the standard DOM storage (localStorage).
What I've observed is that while inserting the dataset with this method, Chrome's memory usage goes up to around 2GB. Also, because saving to localStorage is synchronous, this blocks the DOM and the app becomes unresponsive for 30 seconds.
I thought it may be a Lawnchair issue, so I tried switching now to localForage using IndexedDB. This has had 2 great improvements:
- localForage is asynchronous so the app remains responsive
- Memory usage doesn't climb at all
However, it takes about 10 minutes to write the data to indexed db. The first 5 minutes, CPU sits at around 30% and nothing is written to indexed db. The last 5 minutes CPU goes down and writing commences at a slow pace (maybe 10 records / second).
My question is thus: Are these behaviours normal and if so, how can I store this data (roughly 2Mb) in persistent local storage without it either taking ages or consuming a huge amount of memory?
In case anyone is interested, I've narrowed down the problem to the fact that with localforage I've been trying to insert thousands of key value pairs, with the value being a JSON object. This takes a very long time. Rather call JSON.stringify(obj) first and insert the string as the value - more than 100x faster.