Intermittent Syslog Messages Missing with Log4j2

Question

I am currently working on a project where I have to manage logging using Log4j2. I am facing a specific issue related to updating log levels in the log4j2.xml file via a PUT API call.

While the API usually returns success or failure as expected, I have noticed that sometimes Syslog messages (such as "updated info to debug") are not generated.

I have enabled the debug option with in the configuration file, but I am not seeing any logs that could help identify the problem.

From the Log4j2 documentation, I understand that during reconfiguration, two Configuration objects will exist until all Loggers have been redirected to the new Configuration, which is then stopped and discarded.

However, what is not clear to me is why some syslog messages are not being sent and how I can troubleshoot this issue effectively. I have checked for potential network issues, but everything seems to be working fine on that front.

Has anyone experienced similar issues with missing syslog messages when using Log4j2? What steps can I take to ensure that all messages are consistently sent and logged? Any guidance or suggestions on how to debug this issue further would be immensely appreciated.

Here is a snippet of my log4j2.xml configuration related to Syslog:

<Syslog name="localSyslog" host="x.x.x.x" port="xxxx" protocol="UDP" facility="user" connectTimeoutMillis="10000" reconnectionDelayMillis="5000"></Syslog>

<Async name="asyncLogAppender">
    <AppenderRef ref="RollingFile"/>
</Async>

<Async name="asyncSysLogAppender">
    <AppenderRef ref="localSyslog"/>
</Async>

<Loggers>
    <Logger name="syslog-logger" level="info" additivity="false">
        <AppenderRef ref="asyncSysLogAppender" />
    </Logger>
    <!-- other configurations go here -->
</Loggers>

Answer 1

You have a complex configuration. The simplified path of a log event is:

1. Logger,
2. LoggerConfig (corresponds to the <Logger> element in the XML), followed by AsyncAppender (corresponds to the <Async> element in the XML),
3. SyslogAppender (corresponds to the <Syslog> element in the XML),
4. SyslogManager
5. Your Syslog server listening on an UDP port.

There are multiple possible sources of unreliability:

• You use UDP between 4 and 5, which is not a reliable protocol. Try switching to TCP.
• The async appender in 2 communicates with the syslog appender in 3 through a queue. If the queue is full, when the appender is stopped events could be dropped (cf. this comment in the source code) and a warning is printed in the status logger. You should set the status logger level at WARN at least and you can try increasing the queue size from the default of 1024:

  <Configuration status="WARN">
    <Appenders>
      <Async name="asyncSysLogAppender" bufferSize="2048">
        <AppenderRef ref="localSyslog"/>
      </Async>
      ...
    </Appenders>
    ...
  </Configuration>
  

  and check for warnings on System.err.
• When you reconfigure Log4j Core, the components in 2 and 3 are replaced with new ones and a ReliabilityStrategy decides what to do with events that remain in the old pipeline. You can change the reliability strategy by setting the Java system property log4j2.reliabilityStrategy to one of "AwaitCompletion" (default), "AwaitUnconditionally" or "Locking". See the Javadoc of ReliabilityStrategy implementations for details.