Cassandra throwing OutOfMemory error on startup

Question

I've got a cassandra node that's just died with an OOM. It had only been up for a few minutes. I've got the min/max heap size set to 8GB on a 16GB box.

Given the recommendation is to not set the JVM heap to more than 8GB I'm reluctant to 'solve' this with increasing that.

Anyone got any clue what might be going on here? I've not got a heap dump unfortunately, although I will try and repro and get one.

ERROR [SharedPool-Worker-1] 2017-09-02 10:39:30,762 JVMStabilityInspector.java:137 - JVM state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_91]
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_91]
    at org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:397) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:208) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:203) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:126) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:114) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:134) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1799) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466) ~[apache-cassandra-3.6.jar:3.6]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_91]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.6.jar:3.6]
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.6.jar:3.6]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]

Update

I could never replicate this unfortunately. I sort of suspect it was environment-related. We were running our Cassandra instance on a very poor performing SAN and I suspect it couldn't flush to disk fast enough or something. A total guess.