How to send data in log file to MongoDB using Telagraf?

Question

Telegraf agent seems not to send data to MongoDB

I am trying to collect log file data like Apache-Tomcat's access_log, app_log, etc using Telegraf and send it to MongoDB 6.x version to store data.

So, these are what I did.

1. downloaded Telegraf lastest version source files from Git,
2. built using "GO" on Linux box and installed MongoDB 6.x version on Windows box like below :

• Telegraf Agent, Apache-Tomcat : Linux box
• MongoDB 6.x version : Windows box

---

3. generated sample access_log file and configured telegraf.conf file like below :

access_log file

0:0:0:0:0:0:0:1 - - [24/Oct/2022:15:57:02 +0530] "GET /manager/html HTTP/1.1" 401 2538
0:0:0:0:0:0:0:1 - admin [24/Oct/2022:15:57:10 +0530] "GET /manager/html HTTP/1.1" 200 15829
0:0:0:0:0:0:0:1 - admin [24/Oct/2022:15:57:10 +0530] "GET /manager/images/tomcat.gif HTTP/1.1" 200 2066
0:0:0:0:0:0:0:1 - admin [24/Oct/2022:15:57:10 +0530] "GET /manager/images/asf-logo.gif HTTP/1.1" 200 7279

telegraf.conf

[agent]
debug = true
quiet = false
logtarget = "file"
logfile = "/app/telegraf/telegraf-master/logs/telegraf.log"

[[inputs.tail]]
files = ["/logs/weblog/apache-tomcat/access_log/access_log"]

[[outputs.mongodb]]
dsn = "mongodb://192.168.0.1:27017"
database = "test"
granularity = "seconds"
authentication = "SCRAM"
username = "admin"
password = "password"

---

4. And started Telegraf agent with telegraf command like below :

start command

./telegraf --config ./telegraf.conf 

@reference https://github.com/influxdata/telegraf @reference https://github.com/influxdata/telegraf/tree/master/plugins/inputs/tail @reference https://github.com/influxdata/telegraf/tree/master/plugins/outputs/mongodb

---

I thought that Telegraf agent would create timeseries collection and send data by doing so. Ultimately, I expected I could find data from specific collection.

However, I couldn't found any data.

So, I want know these :

1. How to check root cause of this issue
2. How to configure target collection on telegraf.conf without creating

Answer 1

You need to tell telegraf how to parse these log files. By default, the tail input will expect data in line protocol, which I doubt your apache logs are in :) You will want to look into the grok parser for example to read the logs and build metrics you want to send to mongodb.