Error accessing Azure Service Bus from Storm cluster

Question

We are trying to read messages from Azure Service Bus (ASB) through Apache Storm cluster using a custom built spout java code. When the storm topology is submitted to run in cluster mode, we are facing the following issues:

• Service or property not registered: com.microsoft.windowsazure.services.servicebus.ServiceBusContract class com.sun.jersey.api.client.Client
• Unable to add Azure Service Bus Connector
• Setup of Services Failed.
• Async loop died!

When the same topology is submitted in local mode (without cluster), the same code works fine and able to receive messages from ASB.

Did anyone face similar issue when accessing ASB from java environment and able to resolve this issue?

Answer 1

See the workaround on https://github.com/Azure/azure-sdk-for-java/issues/466. There are problems with shaded jars, you only need to include a new transformer in the pom.xml to solve the issue. It worked for me.

Answer 2

In the gradle build script I have added this plugin

apply plugin: 'com.github.johnrengelman.shadow'

Instead of jar task I used shadowJar task where I gave archive name and main class. The method mergeServiceFiles() merges the duplicate service file names.

shadowJar {
archiveName = "sample.jar"
manifest {
attributes 'Main-Class': ' com.test.asset.Test'
}
mergeServiceFiles()
}

I hope this helps some one

Thanks, Veeresh

Answer 3

I'm making a jar-with-dependencies of a custom azure-service-bus client and in my case (Carlos/Robert)/issue#466 workaround (maven-shade-plugin) doesn't work.

I always get the same error :

Service or property not registered: com.microsoft.windowsazure.services.servicebus.ServiceBusContract class com.sun.jersey.api.client.Client

NB: I just found another open ticket that is maybe related to the same issue here : issue#465

Root cause of this error is the following wrong resource :

META-INF/services/com.microsoft.windowsazure.core.Builder$Exports

Both azure-core and azure-servicebus (0.8.3) dependencies use the same file (same name and same location) which embed "package to export" list. This 2 files should be merged.

My sad workaround is to use 2 executions of the maven-assembly:

• step A) first execution makes a dependencies directory,
• step B) second execution append my version of the fixed Builder$Exports resource, remove MS signed files(security issue), and make a final jar file.

This solution is not yet acceptable because the fixed resource is not picked from azure dependencies. I had my own extra resource. But today I have no other workaround.

The fixed MS merge result file (src/main/resources) is:

file name

META-INF/services/com.microsoft.windowsazure.core.Builder$Exports

file content

com.microsoft.windowsazure.services.servicebus.Exports
com.microsoft.windowsazure.services.servicebus.implementation.Exports
com.microsoft.windowsazure.core.pipeline.apache.Exports
com.microsoft.windowsazure.core.pipeline.jersey.Exports
com.microsoft.windowsazure.core.utils.Exports
com.microsoft.windowsazure.credentials.Exports
com.microsoft.windowsazure.management.configuration.Exports

pom plugin definitions:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <executions>
                    <execution>
                        <id>dir-jar-with-dependencies</id><!-- step A -->
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <descriptors>
                                <descriptor>src/assembly/dir_jardeps.xml</descriptor>
                            </descriptors>
                            <finalName>busclient</finalName>
                            <appendAssemblyId>false</appendAssemblyId>
                        </configuration>
                    </execution>
                    <execution>
                        <id>jar-with-dependencies-ms-patch</id><!-- step B -->
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <archive>
                                <manifest>
                                    <mainClass>${mainclass}</mainClass>
                                </manifest>
                            </archive>
                            <descriptors>
                                <descriptor>src/assembly/jardeps_mspatch.xml</descriptor>
                            </descriptors>
                            <finalName>busclient</finalName>
                            <appendAssemblyId>false</appendAssemblyId>
                        </configuration>
                    </execution>

file src/assembly/dir_jardeps.xml :

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
    <id>dir-jar-with-dependencies</id>
    <formats>
        <format>dir</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <unpack>true</unpack>
            <scope>runtime</scope>
        </dependencySet>
    </dependencySets>
</assembly>

file src/assembly/dir_jardeps.xml :

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
    <id>dir-jar-with-dependencies</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <fileSets>
        <fileSet>
            <directory>${project.build.directory}/classes/META-INF</directory>
            <outputDirectory>META-INF</outputDirectory>
        </fileSet>
        <fileSet>
            <directory>${project.build.directory}/busclient</directory>
            <outputDirectory>/</outputDirectory>
            <excludes>
             <exclude>META-INF/services/com.microsoft.windowsazure.core.Builder$Exports</exclude>
             <!-- MS jars are signed : remove signatures -->
             <exclude>META-INF/*.SF</exclude>
             <exclude>META-INF/*.DSA</exclude>
             <exclude>META-INF/*.RSA</exclude>
            </excludes>
        </fileSet>
    </fileSets>
</assembly>