At some point my logstash forwarder used to work properly but at some point fail to start with this error message
Failed unmarshalling json: invalid character ']' looking for beginning of value logstash
This message appears only if i run forwarder with this command:
$ /opt/logstash-forwarder/bin/logstash-forwarder -config="/etc/logstash-forwarder"
If i run it as a service it fails without any error message, if i restart the service getting this
Restarting log shipper logstash-forwarder
start-stop-daemon: warning: failed to kill PID: No such process
...done.
The first time stopped forwarder working i found in syslog this message:
connection from UDP: [54.194.184.218]:63707->[10.38.143.181] Nov 21 12:41:38 logstash-forwarder[PID]: 2014/11/21 12:41:38.692517 Read error looking for ack: EOF Nov 21 12:41:38 logstash-forwarder[PID]: 2014/11/21 12:41:38.692592 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt Nov 21 12:41:38 logstash-forwarder[PID]: 2014/11/21 12:41:38.692827 Connecting to X.X.X.X:5000 (X.X.X.X) Nov 21 12:41:38 logstash-forwarder[PID]: 2014/11/21 12:41:38.694128 Failure connecting to X.X.X.X: dial tcp X.X.X.X:5000: connection refused
At server side logstash and ES cluster seems to work fine..
I found many similar problems with mine but not any solution.