Jgroup and weblogic: Sender not in table

Question

I'm trying to implement ehcache replication for my application. Following are the jar versions : ehcache-jgroupsreplication:1.7 ehcache-core 2.5.2 jgroups 3.1.0

When starting my application, getting following line in server logs:

GMS: address=ABC111-33601, cluster=EH_CACHE, physical address=10.x.x.xx:1123

And getting the following warning in application logs:

ABC111-33601: dropped message 1 from ABC222-40262 (sender not in table [ABC111-33601]), view=[ABC111-33601|0] [ABC111-33601]

The echache.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd"
    updateCheck="false">
    <diskStore path="java.io.tmpdir"/>

    <cacheManagerPeerProviderFactory
        class="net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory"
        properties="connect=TCP(bind_port=1123):
        TCPPING(initial_hosts=ABC111[1123],ABC222[1123],ABC333[1123];port_range=10;timeout=3000;num_initial_members=4):
        VERIFY_SUSPECT(timeout=1500):
                        pbcast.NAKACK(use_mcast_xmit=false;use_mcast_xmit_req=false;retransmit_timeout=3000):
                        pbcast.GMS(join_timeout=5000):
                        FRAG2(frag_size=60K)"
        propertySeparator="::" />
    <defaultCache
        maxElementsInMemory="1000"
        eternal="false"
        timeToIdleSeconds="120"
        timeToLiveSeconds="120"
        overflowToDisk="false"
        diskPersistent="false"
        diskExpiryThreadIntervalSeconds="120">              
       </defaultCache>
<cache name="com.abc.tariff"
        maxElementsInMemory="1000"
        eternal="false"
        overflowToDisk="false"
        timeToIdleSeconds="1800"
        timeToLiveSeconds="1800">
        <cacheEventListenerFactory
        class="net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory"
        properties="replicateAsynchronously=true, replicatePuts=true, replicateUpdates=true, replicateUpdatesViaCopy=false, replicateRemovals=true" />      
        </cache>
    <cache name="com.abc.customer"
        maxElementsInMemory="1000"
        eternal="false"
        overflowToDisk="false"
        timeToIdleSeconds="120"
        timeToLiveSeconds="180">
        <cacheEventListenerFactory
        class="net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory"
        properties="replicateAsynchronously=true, replicatePuts=true, replicateUpdates=true, replicateUpdatesViaCopy=false, replicateRemovals=true" />      
        </cache>
</ehcache>

ABC111, ABC222 and ABC333 are not in weblogic cluster.

Any idea why the warning is coming and my guess is that the replication has not started due to this or has it?

Answer 1

Your cluster has not formed.

The warning says that you received a message from ABC222 which claimed to be in the same cluster but wasn't in your view of the cluster.

Your config looks weird anyway, for instance, UNICAST is missing, you have no failure detection protocols, no merge protocols etc. Is this the default JGroups config ehcache ships with? That would be very wrong!

You can use probe (check the JGroups manual for details) to find out if the cluster formed correctly. My guess is that adding a correct bind_addr to TCP would fix the issue here...

Answer 2

After your suggestion I have changed the settings like following:

<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory"
properties="connect=TCP(bind_port=1123;bind_addr=10.x.x.58):
TCPPING(initial_hosts=10.x.x.58[1123],10.x.x.59[1123];port_range=10;timeout=3000;num_initial_members=2;break_on_coord_rsp=true):
MERGE2(min_interval=10000;max_interval=30000):
FD_SOCK:
FD(timeout=3000;max_tries=3):
VERIFY_SUSPECT(timeout=1500):
BARRIER:
pbcast.NAKACK2(use_mcast_xmit=false;discard_delivered_msgs=true):
UNICAST:
pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=4M):
 pbcast.GMS(print_local_addr=true;join_timeout=5000;view_bundling=true):
UFC(max_credits=2M;min_threshold=0.4):
MFC(max_credits=2M;min_threshold=0.4):
FRAG2(frag_size=60K):
pbcast.STATE_TRANSFER"
propertySeparator="::" />

Also added following to weblogics arguments:

-Djava.net.preferIPv4Stack=true -Djgroups.resolve_dns=true -Djgroups.bind_addr=10.x.x.58 -Djgroups.tcpping.initial_hosts=10.x.x.58[1123],10.x.x.59[1123]

Tried runningg the prob as well :

java -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -classpath -cp E:/jgroup/jgroups-3.1.0.Final.jar org.jgroups.tests.Probe

-- send probe on /224.0.75.75:7500

1 (217 bytes):
local_addr=ABC111-65460 [XX-78af-fb20-ae58-XX]
cluster=EH_CACHE
physical_addr=ABC111.qaoneadr.local:1123
view=[ABC222-23806|1] [ABC222-23806, ABC111-65460]
version=3.1.0.Final

2 (247 bytes):
local_addr=ap-insight2t-47964 [bbc6c770-e344-ceaa-18a9-f690284ca154]
cluster=OSCacheBus_insight_II_Insight_SITE_cluster
view=[ap-insight3t-30967|697] [ap-insight3t-30967, ap-insight2t-47964]
physical_addr=10.XX.XX.32:7900
version=3.3.5.Final

3 (217 bytes):
local_addr=ABC222-23806 [XX-d700-a732-227a-XX]
cluster=EH_CACHE
physical_addr=ABC222.qaoneadr.local:1123
view=[ABC222-23806|1] [ABC222-23806, ABC111-65460]
version=3.1.0.Final

3 responses (3 matches, 0 non matches)

But still getting :

ABC111-65460: dropped message 1 from ABC222-23806 (sender not in table [ABC111-65460]), view=[ABC111-65460|0] [ABC111-65460]