Convert JSON Lines to JSON array using Apache Nifi

Question

I have a file that includes (schemaless) JSON Lines encoded data.

For example:

{"foo" : "abc", "bar" : "def" }
{"foo" : "xyz" }
{"foo" : "ghi", "bar" : "jkl", "name" : "The Dude"}

I would like to use NIFI to convert this into a JSON array:

[{"foo" : "abc", "bar" : "def" },{"foo" : "xyz" },{"foo" : "ghi", "bar" : "jkl", "name" : "The Dude"}]

Answer 1

The easiest way to accomplish this in Apache NiFi is to use two ReplaceText processors. In the first, configure as:

• Search Value: \}\s*\{
• Replacement Value: \},\{
• Replacement Strategy: Regex Replace
• Evaluation Mode: Entire Text

This will remove the line breaks between the tuples and insert commas between them. In the second:

• Search Value: (^.*$)
• Replacement Value: [$1]
• Replacement Strategy: Regex Replace
• Evaluation Mode: Entire Text

This will add the enclosing brackets around the JSON array. There are other ways to accomplish this with ExecuteScript or JoltTransformJSON processors, but they are more complicated and brittle.