parse recursively nested Json structure with Play framework

Question

we are using Play framework 2.3.4. From one of the APIs we make a web service call to a third party service - structure of the returned response is dynamic and may change. Only sub-structure that's static within the response JSON is a particular element and nesting inside it. for e.g.

{
 "response": 
  {
   "someElement1": "",
   "element2": true,
   "buckets": [
     {
       "key": "keyvalue",
       "docCount": 10,
       "somethingElse": {
         "buckets": [
           {
             "key": "keyvalue1",
             "docCount": 5,
             "somethingElseChild": {
               "buckets": [
                 {
                   "key": "Tan",
                   "docCount": 1
                 }
               ]
             }
           },
           {
             "key": "keyvalue2",
             "docCount": 3,
             "somethingElseChild": {
               "buckets": [
                 {
                   "key": "Ban",
                   "docCount": 6
                 }
               ]
             }
           }
         ]
       }
     }
   ]
  }
}

we don't know how the response structure is going to look like but ONLY thing we know is that there will be "buckets" nested elements somewhere in the response and as you can see there are other nested "buckets" inside a top level "buckets" element. also please note that structure inside buckets array is also not clear and if there will be another sub bucket it's definite that sub bucket must be somewhere inside parent bucket - so that pattern is consistent.

what's the best way to parse such recursive structure and populate following Bucket class recursively?

case class Bucket(key:String,docCount, subBuckets: List[Bucket] ) 

First I was thinking to

val json = Json.parse(serviveResponse)
val buckets = (json \ "response" \\ "buckets") 

but that will not bring bring buckets recursively and not right way to traverse.

Any ideas?

Answer 1

To make a Reads[T] for a recursive type T, you have to

• define it as a lazy val,
• use lazyRead where you need recursive parsing,
• and manually pass to it the Reads[T] object or its derivative.

Of course you have to know what paths exactly the buckets element may appear at, and also account for it missing from any of those. You can use orElse to try several paths.

For your definition of Bucket, the Reads may look like this:

import play.api.libs.json._
import play.api.libs.functional.syntax._

implicit lazy val readBucket: Reads[Bucket] = (
  (__ \ "key").read[String] and
  (__ \ "docCount").read[Int] and
  (
    (__ \ "somethingElse" \ "buckets").lazyRead(Reads.list(readBucket)) orElse
    (__ \ "somethingElseChild" \ "buckets").lazyRead(Reads.list(readBucket)) orElse
    Reads.pure(List.empty[Bucket])
  )
) (Bucket.apply _)

You can simplify it a bit by extracting the common part to a function, e.g.:

def readsBucketsAt(path: JsPath): Reads[List[Bucket]] =
  (path \ "buckets").lazyRead(Reads.list(readBucket))

/* ... 
  readsBucketsAt(__ \ "somethingElse") orElse
  readsBucketsAt(__ \ "somethingElseChild") orElse
  Reads.pure(List.empty[Bucket])
... */

This example doesn't account for possible merging of several buckets arrays at different paths inside a single bucket. So if you need that functionality, I believe you'd have to define and use a play.api.libs.functional.Monoid instance for Reads[List[T]], or somehow combine the existing monoid instances for JsArray.

Answer 2

Parse recursively. Something like this (not tested):

case class Bucket(key: String, count: Int, sub: List[Bucket])

def FAIL = throw new Exception  // Put better error-handling here

def getBucket(js: JsValue): Bucket = js match {
  case o: JsObject =>
    val key = (o \ "key") match {
      case JsString(s) => s
      case _ => FAIL
    }
    val count = (o \ "docCount") match {
      case JsNumber(n) => n.toInt
      case _ => FAIL
    }
    val sub = (o \ "buckets") match {
      case a: JsArray => a.value.toList.map(getBucket)
      case _ => Nil
    }
    Bucket(key, count, sub)
  case _ => throw new Exception
}