Logstash conf file for parsing django exceptions

Question

I have been trying to use logstash, elastic search, and Kibana for monitoring my django server. I have set the conf file as given below

input {
   tcp { port => 5000 codec => json }
   udp { port => 5000 type => syslog }
}

output {
  elasticsearch_http {
  host => "127.0.0.1"
  port => 9200
}
stdout { codec => rubydebug }

} But the messages logged are too lengthy and could not find a method to parse it. Any help is appreciated

Answer 1

As far as I can tell, there is not a pattern or built-in that will directly parse Django exceptions.

You need to tell the forwarding agent to target the Django log files that you're generating, marking them as "type": "django".

Then, on the Logstash server, you can use the following:

pattern:

DJANGO_LOGLEVEL (DEBUG|INFO|ERROR|WARNING|CRITICAL)
DJANGO_LOG %{DJANGO_LOGLEVEL:log_level}\s+%{TIMESTAMP_ISO8601:log_timestamp}\s+%{TZ:log_tz}\s+%{NOTSPACE:logger}\s+%{WORD:module}\s+%{POSINT:proc_id}\s+%{GREEDYDATA:content}

filter:

filter {
     if [type] == "django" {
        grok {
             match => ["message", "%{DJANGO_LOG}" ]
        }

        date {
            match => [ "timestamp", "ISO8601", "YYYY-MM-dd HH:mm:ss,SSS"]
            target => "@timestamp"
        }
     }
}

if you don't want to add the pattern file, you can expand the DJANGO_LOGLEVEL pattern into the %{DJANGO_LOGLEVEL:log_level} field and place the targeting rule that follows DJANGO_LOG into the grok match placeholder.