Why is Array.push(element) not working sometimes?

Question

I'm using node.js for a project, and I have this certain structure in my code which is causing problems. I have an array dateArr of sequential dates that contains 106 items. I have an array resultArr to hold resulting data. My code structure is like this:

function grabData(value, index, dateArr) {
  cassandra client execute query with value from dateArr {
    if (!err) {
      if (result has more than 0 rows) {
        process the query data
        push to resultArr
      }
      if (result is empty) {
        push empty set to resultArr
      }
    }
  }
}

dateArr.forEach(grabData);

I logged the size of resultArr after each iteration and it appears that on some iterations nothing is being pushed to resultArr. The code completes with only 66 items stored in resultArr when 106 items should be stored because the I/O structure between dateArr and resultArr is 1 to 1.

Answer 1

I logged the size of resultArr after each iteration

When the grabData method gets called you start a query to somewhere, or someone named cassandra. As Felix Kling wrote, your notation seems to show an asynchronous function, that starts the request and returns.

As the function is asynchronous, you don't know when the query is ready. That might even take very long, when the database is locked for a dump, or whatever.

When you return from grabData "iteration" and check your resultArr, the resultArr will exactly be filled with each returned value. It might even be that the fifth iteration returns a query before the third, or fourth or any iteration before. So in you resultArr you sometimes have values of iteration n at some point m<n or o>n.

As long as you (or we) don't know anything about how cassandra operates, you cannot say when a query gets answered.

So when you check your result array, it returns the number of completed queries, not the number of iterations.

Answer 2

Found the root cause: There is a hard limit when querying Cassandra using node.js. The query that I am trying to completely execute is too large. Breaking dateArr up into smaller chunks and querying using those smaller pieces solved the problem.