StreamParse: IOError: Local port: 6627 already in use, unable to open ssh tunnel to nimbus.server.local:6627

Question

Setup:

• Storm 0.10.0
• Streamparse 2.1.4
• Centos 6.5
• Python 2.7 (Streamparse needs it)

(Yes i know they are outdated, however i couldnt get anything working with Storm 1.0, its just broken with streamparse 3)

When I attempt to launch a "streamparse submit" from either my nimbus server, or another server in my topology I get the following error:

"IOError: Local port: 6627 already in use, unable to open ssh tunnel to nimbus.server.local:6627."

But ofcourse 6627 is in use on my nimbus server? Its the Thrify port. So i tried moving the Thrifty port to 6637 and restarting Nimbus. But I get the same error back from the client submitting it:

IOError: Local port: 6627 already in use, unable to open ssh tunnel to nimbus.server.local:6627.

Even a netstat tuanp shows that 6627 shows that nothing is listening on that port on nimbus or the box executing the submit.

I have a feeling something to do with SSHD config and allowing tunneling, and that isn't being handled properly by Nimbus and giving an incorrect error when trying to establish the tunnel.

Has anyone else experienced this?

Answer 1

This is what I ended up doing to deploy streamparse Storm topology in the local Storm cluster:

> sparse quickstart quickstart-2.1.4
> cd quickstart-2.1.4
> sparse jar
> storm jar _build/quickstart-2.1.4-0.0.1-SNAPSHOT-standalone.jar streamparse.commands.submit_topology topologies/wordcount.clj

This worked with streamparse 2.1.4 and Storm 0.9.5

Answer 2

I got the same error when running storm topology.

I made following changes, then it worked fine, Added the following property

In config.json,

        "use_ssh_for_nimbus": false,
        "use_virtualenv": false,

In fabfile.py,

        from fabric.api import env
        env.use_ssh_config = False
        env.password = '****'
        from streamparse.ext.fabric import *

And submitted as "sparse submit" Please let me know, if it worked, or share the config file