Scodec: Using vectorOfN with a vlong field

Question

I am playing around with the Bitcoin blockchain to learn Scala and some useful libraries. Currently I am trying to decode and encode Blocks with SCodec and my problem is that the vectorOfN function takes its size as an Int. How can I use a long field for the size while still preserving the full value range. In other words is there a vectorOfLongN function?

This is my code which would compile fine if I were using vintL instead of vlongL:

object Block {
  implicit val codec: Codec[Block] = {
    ("header" | Codec[BlockHeader]) ::
    (("numTx" | vlongL) >>:~
      { numTx => ("transactions" | vectorOfN(provide(numTx), Codec[Transaction]) ).hlist })
  }.as[Block]
}

You may assume that appropriate Codecs for the Blockheader and the Transactions are implemented. Actually, vlong is used as a simplification for this question, because Bitcoin uses its own codec for variable sized ints.

Answer 1

I'm not a scodec specialist but my common sense suggests that this is not possible because Scala's Vector being a subtype of GenSeqLike is limited to have length of type Int and apply that accepts Int index as its argument. And AFAIU this limitation comes from the underlying JVM platform where you can't have an array of size more than Integer.MAX_VALUE i.e. around 2^31 (see also "Criticism of Java" wiki). And although Vector theoretically could have work this limitation around, it was not done. So it makes no sense for vectorOfN to support Long size as well.

In other words, if you want something like this, you probably should start from creating your own Vector-like class that does support Long indices working around JVM limitations.

Answer 2

You may want to take a look at scodec-stream, which comes in handy when all of your data is not available immediately or does not fit into memory.

Basically, you would use your usual codecs.X and turn it into a StreamDecoder via scodec.stream.decode.many(normal_codec). This way you can work with the data through scodec without the need to load it into memory entirely.

A StreamDecoder then offers methods like decodeInputStream along scodec's usual decode.

(I used it a while ago in a slightly different context – parsing data sent by a client to a server – but it looks like it would apply here as well).