It doesn't seem possible in Java for this scenario to occur, but here's what I'm seeing in a jstack dump of a deadlocked process:
"clrd-normalpri-13" Id=165 BLOCKED on java.text.SimpleDateFormat@1b9ff8be owned by "clrd-normalpri-12" Id=164
at org.jclouds.date.internal.SimpleDateFormatDateService.rfc822DateParse(SimpleDateFormatDateService.java:103)
- blocked on java.text.SimpleDateFormat@1b9ff8be
at org.jclouds.blobstore.functions.ParseSystemAndUserMetadataFromHeaders.parseLastModifiedOrThrowException(ParseSystemAndUserMetadataFromHeaders.java:99)
at org.jclouds.blobstore.functions.ParseSystemAndUserMetadataFromHeaders.apply(ParseSystemAndUserMetadataFromHeaders.java:72)
at org.jclouds.s3.functions.ParseObjectMetadataFromHeaders.apply(ParseObjectMetadataFromHeaders.java:62)
at org.jclouds.s3.functions.ParseObjectFromHeadersAndHttpContent.apply(ParseObjectFromHeadersAndHttpContent.java:48)
at org.jclouds.s3.functions.ParseObjectFromHeadersAndHttpContent.apply(ParseObjectFromHeadersAndHttpContent.java:34)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
...
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@39133244
"clrd-normalpri-12" Id=164 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f32283d
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f32283d
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The above snippet shows one of 14 threads BLOCKED on an object monitor, followed by the owning thread, which is just waiting in the pool for work to do, seeming to indicate that the owning thread somehow got out of the synchronized block without releasing the lock.
The jclouds (2.4.0) code (relevant portions only) managing the lock looks like this:
public class SimpleDateFormatDateService implements DateService {
...
// @GuardedBy("this")
private static final SimpleDateFormat rfc822SimpleDateFormat = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss 'GMT'", Locale.US);
...
@Override
public final String rfc822DateFormat(Date date) {
synchronized (rfc822SimpleDateFormat) {
return rfc822SimpleDateFormat.format(date);
}
}
...
@Override
public final Date rfc822DateParse(String toParse) {
synchronized (rfc822SimpleDateFormat) {
try {
return rfc822SimpleDateFormat.parse(toParse);
} catch (ParseException pe) {
throw new IllegalArgumentException("Error parsing data at " + pe.getErrorOffset(), pe);
}
}
}
...
}
I've seen perhaps 1 or 2 situations like this before in my career, but was never in a position to track them down, so I always just assumed I misread something somewhere. Now I'm wondering if there are weird cases in Java (1.8.0) where this type of scenario CAN occur.
I can't claim this is a jclouds bug (though I did file this issue: https://issues.apache.org/jira/browse/JCLOUDS-1607), because it looks impossible. Anyone know anything about this?