JClouds: Multiple threads BLOCKED on object monitor owned by thread waiting in pool for work to do

90 Views Asked by At

It doesn't seem possible in Java for this scenario to occur, but here's what I'm seeing in a jstack dump of a deadlocked process:

"clrd-normalpri-13" Id=165 BLOCKED on java.text.SimpleDateFormat@1b9ff8be owned by "clrd-normalpri-12" Id=164
        at org.jclouds.date.internal.SimpleDateFormatDateService.rfc822DateParse(SimpleDateFormatDateService.java:103)
        -  blocked on java.text.SimpleDateFormat@1b9ff8be
        at org.jclouds.blobstore.functions.ParseSystemAndUserMetadataFromHeaders.parseLastModifiedOrThrowException(ParseSystemAndUserMetadataFromHeaders.java:99)
        at org.jclouds.blobstore.functions.ParseSystemAndUserMetadataFromHeaders.apply(ParseSystemAndUserMetadataFromHeaders.java:72)
        at org.jclouds.s3.functions.ParseObjectMetadataFromHeaders.apply(ParseObjectMetadataFromHeaders.java:62)
        at org.jclouds.s3.functions.ParseObjectFromHeadersAndHttpContent.apply(ParseObjectFromHeadersAndHttpContent.java:48)
        at org.jclouds.s3.functions.ParseObjectFromHeadersAndHttpContent.apply(ParseObjectFromHeadersAndHttpContent.java:34)
        at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
        at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
        ...

        Number of locked synchronizers = 1
        - java.util.concurrent.ThreadPoolExecutor$Worker@39133244

"clrd-normalpri-12" Id=164 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f32283d
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f32283d
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

The above snippet shows one of 14 threads BLOCKED on an object monitor, followed by the owning thread, which is just waiting in the pool for work to do, seeming to indicate that the owning thread somehow got out of the synchronized block without releasing the lock.

The jclouds (2.4.0) code (relevant portions only) managing the lock looks like this:

public class SimpleDateFormatDateService implements DateService {
   ...
   // @GuardedBy("this")
   private static final SimpleDateFormat rfc822SimpleDateFormat = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss 'GMT'", Locale.US);
   ...
   @Override
   public final String rfc822DateFormat(Date date) {
      synchronized (rfc822SimpleDateFormat) {
         return rfc822SimpleDateFormat.format(date);
      }
   }
   ...
   @Override
   public final Date rfc822DateParse(String toParse) {
      synchronized (rfc822SimpleDateFormat) {
         try {
            return rfc822SimpleDateFormat.parse(toParse);
         } catch (ParseException pe) {
            throw new IllegalArgumentException("Error parsing data at " + pe.getErrorOffset(), pe);
         }
      }
   }
   ...
}

I've seen perhaps 1 or 2 situations like this before in my career, but was never in a position to track them down, so I always just assumed I misread something somewhere. Now I'm wondering if there are weird cases in Java (1.8.0) where this type of scenario CAN occur.

I can't claim this is a jclouds bug (though I did file this issue: https://issues.apache.org/jira/browse/JCLOUDS-1607), because it looks impossible. Anyone know anything about this?

0

There are 0 best solutions below