In which format to record/store numerical data for further JavaScript processing?

Question

I'm planning to collect some data over a few months (daily) for further processing and representation in JavaScipt (probably using any js libraries, such as d3.js, etc. I don't know which one yet). The data will consist of:

• date
• one integer
• one decimal number

Which file/data format would you recommend for recording data for subsequent work with JavaScript?

Answer 1

I think CSV would be more appropriate here because it sounds like it's just going to be a big long list of datapoints.

JSON would work, just like XML or any other system would work, but as much as I love JSON, it is not necessarily well-suited to the job:

• JSON will be space-inefficient. It requires lots of punctuation characters.

• JSON will be memory-inefficient. To add anything to the JSON file you'll need to:

  1. Read the entire file as one long string
  2. Parse that long string as JSON, saving it in memory as a big hash or array
  3. Insert your new data
  4. Re-encode the big hash or array as one long string
  5. Overwrite your old file with that new long string
• JSON will be harder to read... unless it's stored in "pretty" format, in which case it'll be even more space-inefficient.

CSV requires much less superfluous punctuation, and you can append a new line of data to the end of your file without needing to read and write the entire thing.

Consider:

JSON (not pretty):

{"01012016":[42,0.8675309],"01022016":[12,9.87654321]}

JSON (pretty):

{
  "01012016":[
    42,
    0.8675309
  ],
  "01022016":[
    12,
    9.87654321
  ]
}

CSV:

01012016,42,0.8675309
01022016,12,9.87654321

Javascript doesn't have a built-in CSV parser in the way it has JSON.parse... because parsing CSV is really easy! Here's just one of many ways of doing it:

var splitByLine = myCSVString.split("\n");
var splitByLineAndComma = [];
splitByLine.forEach(function(line){
  splitByLineAndComma.push(line.split(","));
});

Answer 2

I would default to using JSON, which is not only lightweight, but JavaScript also has a built in, easy-to-use JSON parser: JSON.parse.

JSON is a collection of name/value pairs, which sounds ideal for the use case you've suggested:

[
    {
        "my-date": 1451744353495,
        "my-int": 42,
        "my-decimal": 3.1415926535
    },
    {
        "my-date": 1451744353496,
        "my-int": 43,
        "my-decimal": 2.7182818284
    },
    {
        "my-date": 1451744353497,
        "my-int": 44,
        "my-decimal": 1.4142135623
    }
]

If you have JSON as a string, just pop it into JSON.parse, and you're ready to roll with a shiny new JavaScript object:

var obj = JSON.parse(str);
obj[0]["my-int"] === 42; // true