Circe list deserialization with best-attempt and error reporting

Question

I'm using Circe to deserialize json containing a list. Sometimes a few items in the json list are corrupted, and that causes the entire deserialization to fail. Instead, I want Circe to make a best attempt, and to return a list of all the successfully deserialized list items, together with a list of errors for the corrupted items. How is this best done in Circe?

Specifically, lets say I'm trying to deserialize this:

val json = """{ "params": {
  "playlist": {
      "name": "Sample Playlist",
      "items": [
        {
          "clipId":"xyz", 
          "name":"abc",
          "properties": {
            "cat": "siamese",
            "dog": "spaniel"            
          }
        },
        {
          "clipId":"pqr", 
          "name":"def",
          "properties": {
            "cat": "tabby",
            "dog": "terrier"
          }
        } 
      ]
   }
}}"""

I'm doing this with:

import io.circe.Decoder, io.circe.generic.auto._
import scala.util._   

case class Clip(clipId: String, name: String, dog: String)
implicit val decodeClip: Decoder[Clip] = Decoder.instance { c =>
    for {
      id <- c.get[String]("clipId")
      name <- c.get[String]("name")
      dog <- c.downField("properties").get[String]("dog")
    } yield {
      Clip(id, name, dog)
    }
}

val decodeClipsParam = Decoder[List[Clip]].prepare(
  _.downField("params").downField("playlist").downField("items")
)

def deserializedThing(theJson: String) = io.circe.parser.decode(theJson)(decodeClipsParam)

It works fine, and correctly deserializes:

scala> deserializedThing(json)
res1: Either[io.circe.Error,List[circeLab.circeLab.Clip]] = Right(List(Clip(xyz,abc,spaniel), Clip(pqr,def,terrier)))

But if I now corrupt one single item of the json list (by changing one of the "dog" keys to "doggg" say), then the entire deserialization fails - it doesn't give me the list of uncorrupted Clip items, it just tells me that it failed.

So instead of deserializing into List[Clip] I'd like to deserialize into List[Try[Clip]], where each item is either like Success(Clip(xyz,abc,spaniel)), or Failure(ErrorDescriptionForThatItem).

I was able to achieve this in Argonaut (using some rather ugly code), but can't figure out the syntax in Circe. What's the best way to achieve this? Thanks!

Answer 1

Ok, so this solution works:

import io.circe.{Json, Decoder}
import io.circe.parser.parse 
import scala.util.{Try, Success, Failure} 

// Throw this exception if the list of items can't even be retrieved
case class ParseException(msg: String) extends Exception(msg) 

case class Clip(clipId: String, name: String, dog: String)
// This is the decoder that tries to decode an individual item
implicit val decodeClip: Decoder[Clip] = Decoder.instance { c =>
    for {
      id <- c.get[String]("clipId")
      name <- c.get[String]("name")
      dog <- c.downField("properties").get[String]("dog")
    } yield {
      Clip(id, name, dog)
    }
}


// Turn a string into a json doc
def jsonDoc(str: String) = parse(str).getOrElse(Json.Null)

// Attempt to retrieve the list of json objects appearing in "items"
def getListOfItemsAsJsonObjects(doc: Json): Try[List[Json]] = doc.hcursor.downField("params").downField("playlist").downField("items").focus match {
  case None => Failure(ParseException("Couldn't get items"))
  case Some(obj) => obj.asArray match {
    case None => Failure(ParseException("Couldn't turn to array"))
    case Some(arr) => Success(arr.toList)
  }
}


// Finally, map each json object from Items to a Try[Clip], so individual corrupted items don't affect others.
def tryListOfTries(str: String) = getListOfItemsAsJsonObjects(jsonDoc(str)).map(ok => ok.map(_.as[Clip].toTry))

If there are no corruptions in the json string, then tryListOfTries(json) will return this:

Success(List(Success(Clip(xyz,abc,spaniel)), Success(Clip(pqr,def,terrier))))

If you corrupt an individual item you'll get something like this, with the other items decoded ok:

Success(List(Failure(DecodingFailure(Attempt to decode value on failed cursor, List(DownField(dog), DownField(properties)))), Success(Clip(pqr,def,terrier))))

And if you corrupt something at a higher level so it can't even retrieve the items array, then you'll get a Failure at the top level:

Failure(ParseException: Couldn't get items)

Not sure whether there's a more idiomatic solution, but I couldn't find one so I hope this helps someone.