Processing JSON from a .txt file and converting to a DataFrame in Julia

Question

Cross posting from Julia Discourse in case anyone here has any leads.

I’m just looking for some insight into why the below code is returning a dataframe containing just the first line of my json file. If you’d like to try working with the file I’m working with, you can download the aminer_papers_0.zip from the Microsoft Open Academic Graph site, I’m using the first file in that group of files.

using JSON3, DataFrames, CSV
file_name = "path/aminer_papers_0.txt"
json_string = read(file_name, String)
js = JSON3.read(json_string)
df = DataFrame([js])

The resulting DataFrame has just one line, but the column titles are correct, as is the first line. To me the mystery is why the rest isn’t getting processed. I think I can rule out that read() is only reading the first JSON object, because I can index into the resulting object and see many JSON objects:

enter image description here

My first guess was maybe the newline \n was causing escape issues, and tried to use chomp to get rid of them, but couldn’t get it to work.

Anyway - any help would be greatly appreciated!

Answer 1

I think the problem is that the file is in JSON Lines format, and the JSON3 library only returns the first valid JSON value that it finds at the start of a string unless told otherwise.

tl;dr

Call JSON3.read with the keyword argument jsonlines=true.

Why?

By default, JSON3 interprets a string passed to its read function as a single "JSON text", defined by RFC 8259 section 1.3.2:

A JSON text is a serialized value....

(My emphasis on the use of the indefinite singular article "a.") A "JSON value" is defined in section 1.3.3:

A JSON value MUST be an object, array, number, or string, or one of the following three literal names: false, null, true.

A string with multiple JSON values in it is technically multiple "JSON texts." It is up to the parser to determine what part of the string argument you give it is a JSON text, and the authors of JSON3 chose as the default behavior to parse from the start of the string to the end of the first valid JSON value.

In order to get JSON3 to read the string as multiple JSON values, you have to give it the keyword option jsonlines=true, which is documented as:

jsonlines: A Bool indicating that the json_str contains newline delimited JSON strings, which will be read into a JSON3.Array of the JSON values. See jsonlines for reference. [default false]

Example

Take for example this simple string:

two_values = "3.14\n2.72"

Each one of these lines is a valid JSON serialization of a number. However, when passed to JSON3.read, only the first is parsed:

using JSON3
@assert JSON3.read(two_values) == 3.14

Using jsonlines=true, both values are parsed and returned as a JSON3.Array struct:

@assert JSON3.read(two_values, jsonlines=true) == [3.14, 2.72]

Other Packages

The JSON.jl library, which people might use by default given the name, does not implement parsing of JSON Lines strings at all, leaving it up to the caller to properly split the string as needed:

using JSON
JSON.parse(two_values)
# ERROR: Expected end of input
# Line: 1
# Around: ...3.14 2.72...
#                 ^

A simple way to implement reading multiple values is to use eachline:

@assert [JSON.parse(line) for line in eachline(IOBuffer(two_values))] == [3.14, 2.72]