@DynamicPropertySource does not override KafkaProperties

Question

I am using Kafka from TestContainer, and @DynamicPropertySource does not override the KafkaProperties that I am using in the application code (src/main). KafkaProperties are load in the context even before KafkaContainer. Below you can see my configuration.

KafkaProducerConfig in production code (main).

@Bean
public <T> KafkaProducer<String, T> defaultKafkaProducer(KafkaProperties kafkaProperties) {
    Map<String, Object> props = new HashMap<>();

    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaProperties.getBootstrapServers());
    ...
}

And in the test I have the abstract integration tests with a @Config for the container configuration.

@Log4j2
@AutoConfigureMetrics
@TestPropertySource(locations = "classpath:application-test.properties")
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, classes = {Application.class})
@AutoConfigureWireMock(port = 0)
//@Testcontainers
@Import(KafkaTestContainerConfig.class)
public abstract class AbstractIntegrationTest {
    ...
}

@Configuration
public class KafkaTestContainerConfig {

    @Container
    public static KafkaContainer kafka = new KafkaContainer(
        DockerImageName.parse("confluentinc/cp-kafka:7.2.2.arm64"))
        .withEnv("KAFKA_AUTO_CREATE_TOPICS_ENABLE", "true")
        .withEnv("KAFKA_CREATE_TOPICS", "my-topic")
        .withEmbeddedZookeeper();


    @DynamicPropertySource
    public static void addKafkaProperties(DynamicPropertyRegistry registry) {
        kafka.start();

        registry.add("spring.kafka.bootstrap-servers", () -> "localhost:" + kafka.getMappedPort(9093).toString());
}

When I run/debug a tests, the kafkaProperties.getBootstrapServers() the producer configuration gets the default bootstrap value. Debuggin I saw the defaultKafkaProducer is being executed before the kafka container. I tried to add @TestContainer to the AbstractIntegrationTests (as you can see above), but it does not have an effect, which makes sense because I understand the container lifecycle is integrated in the managed of Spring test lifecycle. I tried to add @Autowired to the DynamicPropertyRegistry and KafkaProperties, without any affect (which would be also expected). I also tried to avoid the KafkaTestContainerConfig class moving the container config to the Abstract Integration Test class, but with the same result I feel I am misunderstanding something with the lifecycle management, but I cannot find the error.

I am using SpringBoot 2.7.X with Spring 5.3.X and JDK 17.

Answer 1

I had the same issue and could solve it by avoiding KafkaAdmin auto configuration and instead creating manually and overwrite bootstrapServers. Not sure if this is the best way, but at least it works for me to run my tests green.

@Bean
public KafkaAdmin kafkaAdmin(KafkaConnectionDetails connectionDetails) {
    connectionDetails.getBootstrapServers().clear();
    connectionDetails.getBootstrapServers().add(this.bootstrapServerUrl);
    return new KafkaAdmin(producerConfig());
}

Answer 2

Finally I fix the problem moving the @DynamicPropertySource and the container starting to the AbstractIntegrationTest class, it seems the annotation has to be in the same class than the SpringBootTest to ensure it is executed in the expected order