I'm building a system test platform with multiple services in Spring Boot 3 with Maven.
I'm using reactive services. Junit, the Failsage plugin, and Testcontainers to run the tests
I run my tests in an API Gateway service module with @SpringBootTest and compile them in realtime the other modules and run the in Testcontainers: (Discovery Service (Netflix Eureka)), Service A and Service B).
I need my tests to run on MacOS and Linux although MacOS is the big challenge.
My problem is how I configure my discovery clients so that both API Gateway and Service A would reach Service B while one is a @SpringBootTest running in the host machine and one is a container using feign but both getting the service's address from Discovery An Illustration of the situation
This is how I start my services:
{
private static GenericContainer initializeServiceContainer(String service, int port, String... entryPoint) {
return new GenericContainer<>(
new ImageFromDockerfile()
.withFileFromPath(".", Path.of("../" + service + "/target")))
.withDockerfileFromBuilder(builder -> builder
.from("amazoncorretto:17-al2023-jdk")
.add(jarFileMap.get(service), "app.jar") // I dynamically get the service jar
.run("sh -c 'touch app.jar'")
.entryPoint(entryPoint)
)).withExposedPorts(port)
.withExtraHost("host.docker.internal", "host-gateway") // host.docker.internal also in Linux
.withAccessToHost(true)
.withCreateContainerCmdModifier(cmd -> cmd // I have to force a static port binding because then the port registered in discovery would be wrong
.withPortBindings(new PortBinding(Ports.Binding.bindPort(port), new ExposedPort(port)))
);
}
Entrypoint for service A and B:
"java", "-Dspring.profiles.active=some-profile-a, some-profile-b",
"-Dspring.data.mongodb.uri=mongodb://localhost:" + MONGO_CONTAINER.getMappedPort(27017),
"-Deureka.client.serviceUrl.defaultZone=http://host.docker.internal:8761/eureka/",
"-Djava.security.egd=file:/dev/./urandom", "-jar", "app.jar"
My different attempts:
1. I tried registering with the container's IP and the service port but then only the containers can reach each other but the Gateway can't reach them because in Mac docker runs in a Linux VM and not directly on the host machine.
Eureka config in each service:
@Configuration
public class EurekaConfig {
@NonNull
Environment environment;
@Bean
public EurekaInstanceConfigBean eurekaInstanceConfig(InetUtils inetUtils) throws UnknownHostException {
EurekaInstanceConfigBean config = new EurekaInstanceConfigBean(inetUtils);
String ip = InetAddress.getLocalHost().getHostAddress();
String port = environment.getProperty("server.port");
config.setNonSecurePort(Integer.parseInt(port));
config.setIpAddress(ip);
config.setPreferIpAddress(false);
return config;
}
}
properties yaml in each service:
eureka:
client:
should-unregister-on-shutdown: true
register-with-eureka: true
instance:
prefer-ip-address: true
lease-renewal-interval-in-seconds: 30
2. I tried running all containers in network mode host to run on the docker host (which means the port binding does not take effect), but then we had the same problem as (1). The Gateway can't reach the docker host as it is running in a VM.
{
private static GenericContainer initializeServiceContainer(String service, int port, String... entryPoint) {
return new GenericContainer<>(
new ImageFromDockerfile()
.withFileFromPath(".", Path.of("../" + service + "/target")))
.withDockerfileFromBuilder(builder -> builder
.from("amazoncorretto:17-al2023-jdk")
.add(jarFileMap.get(service), "app.jar") // I dynamiclly get the service jar
.run("sh -c 'touch app.jar'")
.entryPoint(entryPoint)
)).withExposedPorts(port) // does not take effect
.withNetworkMode("host")
.withCreateContainerCmdModifier(cmd -> cmd // does not take effect
.withPortBindings(new PortBinding(Ports.Binding.bindPort(port), new ExposedPort(port)))
);
}
3. I tried registering as localhost with the service port and adding another localhost to /etc/hosts to each container so it will redirect to the IP of host.docker.internal (the host machine localhost). Then the gateway would reach all services but service A couldn't reach Service B through feign.
my service yaml:
eureka:
client:
should-unregister-on-shutdown: true
register-with-eureka: true
instance:
prefer-ip-address: false
lease-renewal-interval-in-seconds: 30
hostname: localhost
instance-id: localhost:${server.port}
4. I tried registering as localhost with the service port but put all services in the same network and gave it an alias: "localhost". Then the gateway would reach all services but service A couldn't reach Service B through Feign because we can't overwrite localhost and if we try different alias the gateway wouldn't be able to reach the services.
{
private static GenericContainer initializeServiceContainer(String service, int port, String... entryPoint) {
return new GenericContainer<>(
new ImageFromDockerfile()
.withFileFromPath(".", Path.of("../" + service + "/target")))
.withDockerfileFromBuilder(builder -> builder
.from("amazoncorretto:17-al2023-jdk")
.add(jarFileMap.get(service), "app.jar") // I dynamically get the service jar
.run("sh -c 'touch app.jar'")
.entryPoint(entryPoint)
))
.withNetwork(network)
.withNetworkAliases("localhost") // or other alias
.withExposedPorts(port)
.withCreateContainerCmdModifier(cmd -> cmd
.withPortBindings(new PortBinding(Ports.Binding.bindPort(port), new ExposedPort(port)))
);
}
5. I tried registering the services with host: host.docker.internal. Then all services managed to reach each other but for the gateway I had to add to my computer's hosts file host.docker.internal that points to the IP equivalent to localhost.
Compromises
- I can do approach 5 by adding the host to each machine running my tests
- Give up on @SpringBootTest and run API-Gateway also in Testcontainers and configure all services in a single network but then we lose all Spring-Test functionalities like debugging and mocking etc.