Can Filebeat send log to Fluentd? I need config files example

Question

I try to send log messages from Filebeat to Fluentd, using fluent-plugin-beats both running in my docker desktop but Filebeat keep sending error.

2023-11-08 17:02:31 2023-11-08T10:02:31.482Z    ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(async(tcp://127.0.0.1:5044)): dial tcp 127.0.0.1:5044: connect: connection refused
2023-11-08 17:02:31 2023-11-08T10:02:31.482Z    INFO    [publisher_pipeline_output]     pipeline/output.go:145  Attempting to reconnect to backoff(async(tcp://127.0.0.1:5044)) with 11 reconnect attempt(s)

This is my filebat.yml

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  paths:
    - /log/test/*.log
    
output.logstash:
  # The Logstash hosts
  hosts: ["127.0.0.1:5044"]

My fluent.conf

<source>
  @type beats
  metadata_as_tag  
  # port 5044
  # bind localhost
</source>


<match *beat>
  @type copy
  <store>
    @type file
    path /output/test
    add_path_suffix true
    path_suffix ".txt"
  </store>
  <store>
    @type stdout
  </store>
</match> 

I have try changing port from Filebeat to Fluentd in my docker-compose.yml here is my config in docker-compose.

version: "3"
services:
  fluentd:
    container_name: fluentd
    build: 
      context: ./fluentd
      dockerfile: Dockerfile
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - ./fluentd/buffer:/fluentd/buffer
      - ./fluentd/data:/output:rw
    ports:
      - "24224:24224"
      - "24224:24224/udp"
      - "1514:1514/udp"
      - "13543:13543/udp"
      - "5044:5044"
    networks:
      - fluent
  filebeat:
    container_name: filebeat
    build:
      context: ./filebeat
      dockerfile: Dockerfile
    volumes: 
      - ./filebeat/log:/log/test/:rw
    networks:
      - fluent
networks:
  fluent:
    driver: bridge

I tried using Filebeat instead of using Docker Desktop, and got this error instead

2023-11-09T08:15:24+00:00   fluent.error    {"error":"undefined method `[]' for nil:NilClass","message":"unexpected error error=\"undefined method `[]' for nil:NilClass\""}

Answer 1

I would first try to send logs to an output file to see if it works.

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  paths:
    - /log/test/*.log

output.file:
  path: /output/test.log

Answer 2

If you notice to the error log, it says connection refused . The problem is that you are trying to connect two different containers by 127.0.0.1. When you are using docker it does not work like this. Based on your docker-conpose.yml you named the FluentD -> fluentd. So on the other containers in you network you can find it by this name, it is like a DNS naming.

You need to change you're filebeat configuration to this:

hosts: ["fluentd:5044"]

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  paths:
    - /log/test/*.log
    
output.logstash:
  # The Logstash hosts
  hosts: ["fluentd:5044"]