Example multiple servers with scale down

Question

We trying to use ActiveMQ Artemis within our Docker container, but one scenario I am not able to get working. This is probably due to some bad configuration. Any help is appreciated (e.g. example configuration).

Installation:

• Docker instance containing an embedded ActiveMQ Artemis broker and a web application
• The broker has clustering, HA and share store defined
• We start 3 docker instances

Scenario:

• Add 200 messages to the queue in one of the web application
• I can see in the logging that all docker instance are handling the messages (this is as expected)
• Kill one of the docker instances

Outcome of scenario:

• Not all messages are being processed (every message on the queue should result to item in the database)
• When restarting the killed docker instance will not result in every message being processed.

Expected outcome:

• When a node is down that a other node is picking up the messages
• When a node comes online again that is help picking up the messages

Questions:

• HA scale down probably does not work because I kill the server.
• Does this only work with persistence on the file system or should this also work in an RDBMS?

Configuration:

Below is the configuration which is in every Docker instance, only the host name (project-two) and the HA settings (master/slave) differ per docker instance. It could be a typo in below because I removed the customer specific names in the configuration.

<configuration
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns="urn:activemq"
    xsi:schemaLocation="urn:activemq /schema/artemis-server.xsd">
    <core xmlns="urn:activemq:core">

        <security-enabled>true</security-enabled>
        <jmx-management-enabled>true</jmx-management-enabled>
        <management-address>activemq.management</management-address>

        <persistence-enabled>true</persistence-enabled>
        <store>
            <database-store>
                <jdbc-driver-class-name>${artemis.databaseDriverClass}</jdbc-driver-class-name>
                <jdbc-connection-url>${artemis.databaseConnectionUrl}</jdbc-connection-url>
                <jdbc-user>${artemis.databaseUsername}</jdbc-user>
                <jdbc-password>${artemis.databasePassword}</jdbc-password>
                <bindings-table-name>ARTEMIS_BINDINGS</bindings-table-name>
                <message-table-name>ARTEMIS_MESSAGE</message-table-name>
                <page-store-table-name>ARTEMIS_PS</page-store-table-name>
                <large-message-table-name>ARTEMIS_LARGE_MESSAGES</large-message-table-name>
                <node-manager-store-table-name>ARTEMIS_NODE_MANAGER</node-manager-store-table-name>
            </database-store>
        </store>

        <connectors>
            <connector name="netty-connector">tcp://project-two:61617</connector>
        </connectors>

        <acceptors>
            <acceptor name="netty-acceptor">tcp://project-two:61617</acceptor>
        </acceptors>

        <!-- cluster information -->
        <broadcast-groups>
            <broadcast-group name="my-broadcast-group">
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
                <broadcast-period>2000</broadcast-period>
                <connector-ref>netty-connector</connector-ref>
            </broadcast-group>
        </broadcast-groups>

        <discovery-groups>
            <discovery-group name="my-discovery-group">
                <group-address>231.7.7.7</group-address>
                <group-port>9876</group-port>
                <refresh-timeout>10000</refresh-timeout>
            </discovery-group>
        </discovery-groups>

        <cluster-connections>
            <cluster-connection name="my-cluster">
                <connector-ref>netty-connector</connector-ref>
                <retry-interval>500</retry-interval>
                <use-duplicate-detection>true</use-duplicate-detection>
                <message-load-balancing>ON_DEMAND</message-load-balancing>
                <max-hops>1</max-hops>
                <discovery-group-ref discovery-group-name="my-discovery-group"/>
            </cluster-connection>
        </cluster-connections>

        
        <security-settings>

        </security-settings>

        <!-- Settings for the redelivery -->
        <address-settings>
            <address-setting match="#">
                <redelivery-delay>5000</redelivery-delay>
                <max-delivery-attempts>2</max-delivery-attempts>
            </address-setting>
        </address-settings>

        <addresses>

        </addresses>

        <ha-policy>
            <shared-store>
                <slave/>
            </shared-store>
        </ha-policy>

    </core>
</configuration>

Answer 1

Docker makes deploying microservice applications very easy but it has some limitations for a production environment. I would take a look to the ArtemisCloud.io operator that provide a way to deploy the Apache ActiveMQ Artemis Broker on Kubernetes.

The ArtemisCloud.io operator also supports message migration for brokers scale down, see https://artemiscloud.io/docs/tutorials/scaleup_and_scaledown/