Forward windows event logs using nxlog to logstash

Question

I am trying to set up a central logging server. I am using nxlog to send window's security events to a ubuntu server running elasticsearch logstash and kibana, but the log files that nxlog outputs dont look right because logstash can't pase any of the data it just throws it all into "message". I am using Windows 8 (prob update to 10 soon) and wondering what I need to do to parse through the data. I tried grok, but some fields are blank and there are also 2 date/times in the log.

*Note I searched all over google and tried other peoples suggestions, but the log always comes out the same. Even if I try to export it as XML instead of JSON.

nxlog.conf

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension syslog>
    Module xm_syslog
</Extension>

<Extension json>
    Module xm_json
</Extension>    

<Input in>
    Module im_msvistalog  
    Query <QueryList> \
        <Query Id="0"> \
  #         <Select Path="Microsoft-Windows-Sysmon/Operational">*</Select> \
  #         <Select Path="Application">*</Select> \
  #         <Select Path="System">*</Select> \
            <Select Path="Security">*</Select> \
        </Query> \
    </QueryList>
    Exec $Message = to_json(); to_syslog_bsd();
</Input>

<Output out>
    Module      om_tcp
    Host        XXX.XXX.XXX.XXX
    Port        9999

</Output>

<Route 1>
    Path        in => out
</Route>

logstash.conf

input {
        tcp {
                port => 9999
                codec => json
                tags => ["windows","eventlog"]
                type => 'nxlog-json'
        }
}
output {
    elasticsearch {
        host => localhost
    }
}

json received at logstash

"<5>Aug 12 15:45:06 JOE>SMITH.com MSWinEventLog\t5\tSecurity\t1319\tWed Aug 12 15:45:03 2015\t4779\tMicrosoft-Windows-Security-Auditing\t\tN/A\tAudit Success\tJOE.SMITH.com\t12551\tA session was disconnected from a Window Station.\r\n\r\nSubject:\r\n\tAccount Name:\t\tnoob.jwsmith\r\n\tAccount Domain:\t\tITORG\r\n\tLogon ID:\t\t0x151258A\r\n\r\nSession:\r\n\tSession Name:\t\tRDP-Tcp#66\r\n\r\nAdditional Information:\r\n\tClient Name:\t\tJOESMITH\r\n\tClient Address:\t\tXXX.XXX.XXX.XXX\r\n\r\n\r\nThis event is generated when a user disconnects from an existing Terminal Services session, or when a user switches away from an existing desktop using Fast User Switching.\n"

Answer 1

Check out our solution for ELK-as-a-Service and the configuration we have for nxlog.

We don't use the json module because there are some issues with nxlog json and logstash. We send the data as text and parse it using logstash capabilities on the other end (in the server)

I'd be happy to help you with this issue.

(Disclaimer - I am the VP Product for logz.io)

This is an example of the configuration we use:

define ROOT C:\\Program Files (x86)\\nxlog
define ROOT_STRING C:\\Program Files (x86)\\nxlog
define CERTDIR %ROOT%\\cert
Moduledir %ROOT%\\modules
CacheDir %ROOT%\\data
Pidfile %ROOT%\\data\\nxlog.pid
SpoolDir %ROOT%\\data
LogFile %ROOT%\\data\\nxlog.log
<Extension charconv>
    Module xm_charconv
    AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
</Extension>
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
    Module im_msvistalog
#Uncomment im_mseventlog for Windows XP/2000/2003
#Module im_mseventlog

    Exec if $raw_event =~ /^#/ drop();
    Exec convert_fields("AUTO", "utf-8");
    Exec    $raw_event = '[<YOUR-TOKEN>][type=msevent]' + $raw_event;
</Input>
<Output out>
    Module  om_tcp
    Host    listener.logz.io
    Port    8010
</Output>
<Route 1>
    Path eventlog => out
</Route>