Maven/Java/Helidon MP: Log4j2.xml <JSONLayout> causing conflict with custom annotation processor

Question

I have a multi module maven project. Module A has an annotation processor, and Module B has a dependency on module A in its pom.xml. Module B also has a src/main/resources/META-INF/services/javax.annotation.processing.Processor file which contains the name of the annotation processor from Module A.

The entire project utilizes slf4j as the logging facade (so I can annotate any class with @Slf4j and call log.info in my code).

I also have a log4j2.xml file under src/main/resources which logs to console. Up until this point, my project compiles and runs as expected.

Now the issue arises when I add a Kafka Appender to my log4j2.xml, and use the tag as shown below:

<Configuration status="info">
    <Appenders>
        <Kafka name="KafkaAppender" topic="123"
               syncSend="false">
                <JSONLayout locationInfo="true" properties="true" charset="ISO-8859-1" compact="true">
                    <KeyValuePair key="profile" value="1"/>
                    <KeyValuePair key="service" value="2"/>
                    <KeyValuePair key="server" value="1234"/>
                </JSONLayout>
            <Property name="bootstrap.servers"
                      value="9092"/>
            <Property name="client.id"
                      value="1223"/>
        </Kafka>
    </Appenders>
    <Loggers>
        <Logger name="org.apache.kafka" level="WARN"/> <!-- avoid recursive logging -->
        <Root level="INFO">
            <AppenderRef ref="KafkaAppender"/>
        </Root>
    </Loggers>
</Configuration>

When I do "mvn clean install" after that, I get the following error:

[ERROR] Bad service configuration file, or exception thrown while constructing Processor object: javax.annotation.processing.Processor: Provider MyProcessor could not be instantiated

I looked into what could possibly cause this, and I noticed if I remove just the <JSONLayout> tag (not the entire <Kafka> tag), the project compiles without error.

I also attemped to use <JSONTemplateLayout> instead of <JSONLayout>, but got the same original error.

Any idea what could be happening here?

Notes:

-My project requires the annotation processor from Module A

-My project requires @Slf4j as the logging facade

Update 1: Seems that specifying any type of layout for <Kafka> tag triggers that error. <CSVLayout> <GelfLayout> <HTMLLayout>, etc doesn't compile either. So the issue doesnt seem to be with <JSONLayout> or <JSONTemplateLayout> specifically.

Update 2: My annotation processor looks like the following:

@SupportedAnnotationTypes("com.my.Config")
@SupportedSourceVersion(SourceVersion.RELEASE_17)
@Slf4j
public class SourceProcessor extends AbstractProcessor {
//other stuff

If I remove the @Slf4j annotation, the original compilation error goes away. This suggests there's an incompatibility between <JSONLayout> from log4j-core maven dependency, and maven lombok dependency. However, @Slf4j annotation is required inside the annotation processor as part of my use case.

Update 3: My pom.xml maven dependencies are as follows:

   <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-api</artifactId>
      <version>2.20.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-core</artifactId>
      <version>2.20.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-slf4j2-impl</artifactId>
      <version>2.20.0</version>
    </dependency>
    <!--Required for SLF4JBridgeHandler-->
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>jul-to-slf4j</artifactId>
      <version>2.0.9</version>
    </dependency>
    <!--Required for @Slf4j annotation-->
    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <version>1.18.30</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>2.0.7</version>
    </dependency>

Answer 1

Since specifying any layout for the <Kafka> tag triggers the error, the issue should not be with the specific layout type, but rather how the inclusion of these layouts impacts the annotation processing classpath or resource loading.

Still, you can temporarily remove the dependency on Module A from Module B and see if the issue persists. That can help confirm if the issue is directly related to the interaction between the annotation processor and the logging configuration.

Make sure the META-INF/services/javax.annotation.processing.Processor file in Module B correctly references the processor class from Module A and that there are no typos or formatting issues.

Consider also creating a minimal version of the annotation processor that logs its initialization process. That can help identify if the processor is being loaded but fails during initialization due to a missing dependency or classpath issue. And increase the verbosity of the Maven build (mvn clean install -X) to get more detailed logs that might point to the exact cause of the issue.

public class MyProcessor extends AbstractProcessor {
    public MyProcessor() {
        super();
        // Minimal logging to confirm instantiation
        System.out.println("MyProcessor instantiated");
    }

    @Override
    public boolean process(Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) {
        // Processor logic here
        return true;
    }
}

Check that your log4j2:

• configuration is not causing classpath issues by accidentally including/excluding necessary classes for the annotation processor. Double-check that your Maven project's resource filtering does not interfere with loading the processor or other classes.
• dependencies (especially those related to Kafka and JSONLayout) are correctly scoped in your Maven configuration. They should be available at compile-time for Module B, which seems to require them, but make sure they do not conflict with the annotation processor's dependencies.
• versions are defined and managed by Maven's dependency management (same for Kafka, and any other related dependencies across your modules). That helps to avoid hidden version conflicts that might be affecting the annotation processor.

---

Update 2: My annotation processor looks like the following:

@SupportedAnnotationTypes("com.my.Config")
@SupportedSourceVersion(SourceVersion.RELEASE_17)
@Slf4j
public class SourceProcessor extends AbstractProcessor {
//other stuff

If I remove the @Slf4j annotation, the original compilation error goes away. This suggests there's an incompatibility between <JSONLayout> from log4j-core maven dependency, and maven lombok dependency. However, @Slf4j annotation is required inside the annotation processor as part of my use case.

Given this new information, the root cause is indeed related to the interaction between the annotation processing environment and the logging configuration, particularly involving the use of Lombok's @Slf4j within the annotation processor.
The compilation error disappearing when you remove the @Slf4j annotation suggests that the annotation processor's attempt to use SLF4J (facilitated by Lombok) conflicts with the project's logging setup when it includes specific layout configurations like <JSONLayout>.

Annotation processors run during the compile phase, which means they operate in a somewhat isolated environment. Adding logging to an annotation processor complicates this because it introduces dependencies that must be resolved during the compile phase.

Lombok's @Slf4j annotation generates SLF4J logging calls at compile time. If your processor uses @Slf4j, it requires SLF4J (and, transitively, Log4j if SLF4J is configured to use Log4j as its backend) to be present on the classpath during compilation.

Configurations like <JSONLayout> require the log4j-core library. The presence of Log4j2-specific configurations could be influencing the classpath or resource loading in a manner that affects the annotation processor, especially when it is trying to use logging via Lombok's @Slf4j.

Make sure the annotation processor has all necessary logging dependencies (including SLF4J and the specific Log4j2 libraries it might indirectly require) correctly scoped and available on the classpath. That might involve adjusting the Maven configuration to explicitly include these dependencies with the appropriate scope for the processor. Examine the output of mvn dependency:tree, mvn help:effective-pom and mvn dependency:analyze.

For dependencies required by your annotation processor, especially during the compile phase (like in the case of annotation processing), make sure they are scoped as provided or compile in your POM. That way, they are available on the classpath during compilation, but are not necessarily included in the final packaged artifact, which is usually the desired behavior for compile-time tools like annotation processors.

For instance:

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>YourVersionHere</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>YourVersionHere</version>
    <scope>provided</scope>
</dependency>

---

As a workaround, instead of relying on Lombok's @Slf4j to introduce logging into your annotation processor, you could try and directly use SLF4J's API. That allows you to manage logging dependencies more explicitly and might avoid the conflict observed with Lombok.

@SupportedAnnotationTypes("com.my.Config")
@SupportedSourceVersion(SourceVersion.RELEASE_17)
public class SourceProcessor extends AbstractProcessor {
    private static final Logger log = LoggerFactory.getLogger(SourceProcessor.class);

    // other stuff
}

---

Given the "pom.xml maven dependencies" from the update 3, the dependencies listed are well-aligned with the requirements for using Log4j2 as the implementation for SLF4J logging and for enabling the @Slf4j annotation via Lombok in your project.

But: You have mixed versions for SLF4J-related dependencies (slf4j-api version 2.0.7 and jul-to-slf4j version 2.0.9). While minor version differences are often compatible, it is best practice to keep these dependencies aligned to the same major and minor version to avoid potential incompatibilities or unexpected behavior. Consider updating these dependencies to the same version to make sure consistency across SLF4J-related components.

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>2.0.9</version> <!-- Updated to match jul-to-slf4j version -->
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>jul-to-slf4j</artifactId>
    <version>2.0.9</version>
</dependency>

Also, make sure the annotation processor (defined in Module A and used in Module B) has access to all necessary logging dependencies during the compilation phase. Since annotation processing occurs at compile time, any dependencies it requires to operate (such as SLF4J and Log4j) must be available on the classpath. See also "Things We Take for Granted #1 − Annotations / How Do Annotation Processors Work?" from Joshua Xu.

If your annotation processor relies on logging, you might need to explicitly declare these logging dependencies in the provided scope within the module that contains the annotation processor. That ensures they are available at compile time but not included in the final packaged artifact, keeping the deployment lightweight.

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.20.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.20.0</version>
    <scope>provided</scope>
</dependency>
<!-- Repeat for other logging dependencies as needed -->

Use the Maven Enforcer Plugin with the dependencyConvergence rule to identify and resolve version conflicts among your dependencies. That rule makes sure all versions of a dependency (or its transitive dependencies) converge to a single version. If there are conflicts, the build fails, prompting you to resolve the issue. That can be particularly useful in complex projects with multiple modules and dependencies.

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-enforcer-plugin</artifactId>
            <version>3.0.0</version>
            <executions>
                <execution>
                    <id>enforce</id>
                    <configuration>
                        <rules>
                            <dependencyConvergence/>
                        </rules>
                    </configuration>
                    <goals>
                        <goal>enforce</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Is there another option to do slf4j logging in the processor?"

As a temporary workaround or even a permanent solution, consider using Java's built-in logging framework, java.util.logging (JUL), for logging within the annotation processor. That approach avoids the complexity of integrating third-party logging frameworks at compile-time. SLF4J provides a bridge to route logs from JUL to SLF4J (even though Andrii Petrivskyi tries to explain said integration in "Using java.util.logging and SLF4J together"), making sure compatibility with your existing logging configuration.

import java.util.logging.Logger;

public class MyProcessor extends AbstractProcessor {
    private static final Logger log = Logger.getLogger(MyProcessor.class.getName());

    // Other processor methods
}

---

Tried everything you suggested, I updated all those dependencies to their latest maven versions as well and tested, same original error.
I cannot set the scope of log4j-api to 'provided' because it throws runtime error java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger. Also I have to use slf4j and log to kafka (cannot log to console), that is the requirement in this project.

Given the project's requirements to use SLF4J for logging and specifically to log to Kafka, consider separating the logging needs of the annotation processor (compile-time) from the application's runtime logging. The goal is to make sure the annotation processor's logging does not interfere with the project's runtime logging requirements, especially since runtime logging to Kafka is a must-have.

• Compile-time Logging: For the annotation processor, use minimalistic logging that does not rely on complex dependencies like SLF4J or Log4j2. Use standard Java mechanisms like System.out.println for outputting essential information or debugging messages during the annotation processing phase. While this approach might seem rudimentary, it is effective in environments where dependency management is critical and potentially conflicting.

• Runtime Logging: Keep your SLF4J and Log4j2 configuration untouched for the application's runtime, as it is required for logging to Kafka. That makes sure the runtime logging setup is isolated from the annotation processing phase and is not affected by the simplifications made for compile-time logging.

For the annotation processor, simplify the logging setup to use minimal dependencies. Since @Slf4j and direct SLF4J usage are causing issues, you might revert to using standard output for critical error messages or debugging information specifically within the annotation processor. While not ideal, this approach avoids the dependency conflict during the compile phase.

import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.Processor;
import javax.lang.model.element.TypeElement;
import java.util.Set;

public class MyProcessor extends AbstractProcessor {
    
    // Use System.out for logging within the processor
    private void log(String message) {
        System.out.println("MyProcessor: " + message);
    }

    @Override
    public boolean process(Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) {
        log("Processing annotations");
        // Your processing logic here
        
        return true; // Return true to claim the annotations were processed
    }
}

That would eliminate dependency-related issues by not relying on external logging frameworks during the compile phase. Compile-time operations (like annotation processing) would not interfere with runtime logging configurations, preserving the ability to log to Kafka at runtime without modification.

Standard output messages (System.out.println) during compilation are typically displayed in the console or the build tool's (e.g., Maven's) output. That means you will see these messages during the build process, which can be sufficient for debugging or informational purposes related to annotation processing.

Since setting the log4j-api dependency scope to provided resulted in a NoClassDefFoundError, it is clear that the runtime environment requires access to log4j-api. That is expected for runtime logging but should not impact compile-time operations like annotation processing. Make sure the runtime environment (where the application is executed) has all necessary logging libraries available on its classpath or module path.

If logging from the annotation processor is essential and must be done through SLF4J, consider programmatically configuring SLF4J within the processor to use a minimal, file-based or console-based logger during the compile phase. That setup can then be overridden by the application's runtime logging configuration.

// Inside your annotation processor
static {
    // Programmatically configure SLF4J to use a simple logger during annotation processing
    System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "debug");
    // Other configuration properties for the simple logger can be set here
}

Answer 2

The Kafka appender has a lot of open issues waiting for user contributions or a new maintainer.

You are probably affected by issue #2066:

• when you compile module B, SourceProcessor is instantiated,
• since your SLF4J logger is static, SLF4J is initialized, followed by Log4j API and the logging implementation on your annotation processor classpath (Log4j Core),
• Log4j Core tries to configure a Kafka appender, which either logs a warning and returns null if you don't specify a layout (which is required) or fails to connect to the Kafka server if you specify a layout and throws an exception (issue #2066),
• in the latter care your compilation fails with an ExceptionInInitializerError, but you don't see it, since the compiler's output is swallowed by your build tool.

Remark: Annotation processors have access to a Messager, which is in my opinion a better way to log events in a processor than SLF4J. The output of Messager is accessible to most build tools.