Best way of parsing huge (10GB+) JSON files

Question

I would like to know what is the best tool, IDE, programming language for parsing data stored as a json file.

I trying pandas in python and ff in R and both of them either crash for memory issues or take too long to process. Do you have experience with them? specially ff?

Is there any good alternative to them?

Answer 1

You can try go's json stream decoder.

Read the file as a stream, and then read token by token, you can decide what to do with each token:

f, err := os.Open("data.json")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

dec := json.NewDecoder(f)
while dec.More() {
    // dec.Token()
}

This should consume very little memory.