Wrong rollback behaviour in a nested transactional context with JTA and DataNucleus

Question

I will first explain my goal, then the symptoms, and then the setup of my environment.

I am working on a project which should finally test the behavior of cumulus4j in a nested transactional context. Therefore I have first written the tests without cumulus4j. The test case is simple: Two objects are written into a database. Every object is written within an own transaction. The two transactions are nested. Now either the main transaction or the nested transaction throws an exception which should cause a rollback of the transaction. The behavior I expect is:

(obj1) main tx success / (obj2) nested tx exception: obj1 in db, obj2 not in db (obj1) main tx exception / (obj2) nested tx success: obj2 not in db, obj2 in db

But I have:

(obj1) main tx success / (obj2) nested tx exception: both objects in db (obj1) main tx exception / (obj2) nested tx success: no object in db

The same tests are running correctly with a plain data source and pure SQL, so I spent a lot of time on checking the JTA and DN configurations.

The transactions are container managed and I followed the instructions on the DataNucleus website. A nested transaction is started by one bean calling an other which is annotated with @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW).

The setup: The whole project is a maven project and should work when switching to the "all-project" and typing "mvn clean install". The test cases are integration tests which call methods of an ejb within a embedded glassfish server. The ejbs are storing the object in a derby database. The relevant settings for datanucleus are:

datanucleus.jtaLocator=custom_jndi
datanucleus.jtaJndiLocation=java:appserver/TransactionManage
javax.jdo.option.TransactionType=JTA
datanucleus.identifierFactory=datanucleus1
datanucleus.storeManagerType=rdbms
javax.jdo.option.ConnectionFactoryName=jdbc/__default
javax.jdo.option.ConnectionFactory2Name=jdbc/__defaultNoXa

The server resource settings in domain.xml are:

<resources>

<jdbc-resource pool-name="DerbyPool" jndi-name="jdbc/__default" />
<jdbc-resource pool-name="DerbyPoolNoXa" jndi-name="jdbc/__defaultNoXa" />
<jdbc-connection-pool name="DerbyPool" datasource-classname="org.apache.derby.jdbc.EmbeddedXADataSource" res-type="javax.sql.XADataSource">
  <property value="${com.sun.aas.instanceRoot}/lib/databases/default" name="databaseName" />
  <property value=";create=true" name="connectionAttributes" />
</jdbc-connection-pool>
 <jdbc-connection-pool name="DerbyPoolNoXa" datasource-classname="org.apache.derby.jdbc.EmbeddedDataSource40" res-type="javax.sql.DataSource">
  <property value="${com.sun.aas.instanceRoot}/lib/databases/default" name="databaseName" />
  <property value=";create=true" name="connectionAttributes" />
  </jdbc-connection-pool>
 </resources>
<servers>
 <server name="server" config-ref="server-config">
  <resource-ref ref="jdbc/__default" />
  <resource-ref ref="jdbc/__defaultNoXa" />
 </server>
</servers>

Can anyone explain the behavior and give me a hint which settings might be wrong?

Regards, Jan