How to configure StatsD and Graphite to run on different servers

Question

I have looked all over for this but have not found anyone talking about how to setup and configure StatsD and Graphite to communicate on separate servers. I currently have everything running on one but I have attempted unsuccessfully to separate them.

Here is how I setup the StatsD exampleConfig.js

exampleconfig
{
  graphitePort: 2003
, graphiteHost: "(graphite server IP)"
, port: 8125
}

The only other thing I can think to setup on the other box is the example-client.py.

currently it says this:
CARBON_SERVER = '127.0.0.1'
CARBON_PORT = 2003

I would think it needs to stay local host to communicate with whisper or graphite on the same server. I have my firewall setup to listen for 2003, and Using a packet dump the server does get the UDP from statsd. It just doesn't seem to get consumed by carbon and graphite.

What am I missing?
Also what is recommended for scaling the statsd graphite setup? I have statsd on its own right now and graphite + carbon + whisper on another server. Does statsd take the most power to run or is it the graphite box? I am wondering this because I will soon be sending millions of bits of data to the servers every day for testing.

Answer 1

Modify example-client.py

If you want to run the example-client.py on a different server that is running your graphite/carbon instance. Then you will need to change the CARBON_SERVER to the IP address of the graphite/carbon server.

---

Network Tests

You might want to also do a few quick tests to make sure that the processes are listening correcting on the ports that your expect and the underlying network will allow this communication.

On the server running graphite/carbon you should be able to check if the server is accepting connections from more than just the localhost via the lsof command

$ lsof -Pi:2003
COMMAND    PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
carbon-ca 1596 graphite   7u  IPv4   9517      0t0  TCP *:2003 (LISTEN)

You can see from the above that I have a carbon-cache process running and listening on all interfaces on TCP 2003.

A very simple test from the remote machine would be to do a telnet connection to the graphite/carbon server on the port that it is listening on ( default: 2003 ) and see if that works.

Example of a listening socket*

$ telnet graphite-server 2003
Trying graphite-server...
Connected to graphite-server.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Example of a closed socket*

$ telnet graphite-server 2003
Trying graphite-server...
telnet: Unable to connect to remote host: Connection refused

Answer 2

So I finally discovered the problem. I was assuming that statsd not only recieved UDP but also sent UDP to carbon. After I realized that statsd sends TCP instead I was able to adjust my firewall and it works great now. I left CARBON_SERVER as local host.

Thanks!

What are the best ways to scale graphite/carbon? Should I separate carbon from graphite? Is that possible? Does Carbon strain the processor more than graphite?

Answer 3

The paperlesspost statsd fork might also be usefull to you: https://github.com/paperlesspost/statsd