Submit topology to storm cluster through streamparse

Question

I am trying to use streamparse to develop and submit the topologies to the storm cluster.

Since streamparse has its default wordcount topology to help user test the cluster, most of the tutorials I could find online is about submitting this default wordcount example to the storm clusters.

My question is how to submit my own topologies? For example, I have a topology named 'mytopology'. Per streamparse's document, I tried

sparse submit --environment prod --name mytopology

and my config file is

{
    "serializer": "json",
    "topology_specs": "topologies/",
    "virtualenv_specs": "virtualenvs/",
    "envs": {
        "prod": {
            "user": "userx",
            "ssh_password": "mypasswd",
            "nimbus": "10.XXX.XX.210",
            "workers": ["10.XXX.XX.206"],
            "log": {
                "path": "/home/userx/stormapp/splog",
                "max_bytes": 1000000,
                "backup_count": 10,
                "level": "info"
            },
            "virtualenv_root": "/home/userx/stormapp/venv"
        }
    }
}

However, the log showed that

JAR created: _build/wordcount-0.0.1-SNAPSHOT.jar

was created and submitted to Nimbus.

Isn't the

--name mytopology

supposed to find the mytopology.py and build something like mytopology.jar and submit that?

Then I checked the project.clj file, the top line is

defproject wordcount "0.0.1-SNAPSHOT"

Now it is confusing. Should I also configure this file? When I do

sparse submit --environment prod --name mytopology

Does it do something that is related to this file? Please help...

Answer 1

I suppose that your have first creating your wordcount project using the following command : sparse quickstart wordcount In this case, "wordcount" will be the name of the topology that will be submitted to Storm using the sparse run command. Now if you want to submit another topology, say mytopology, you have to create another quickstart project called mytopology and edit the config.json file to suit your technical environment. You cannot just copy and rename the "wordcount" project's folder as I guess you've done because "wordcount" appears in your project.clj file.