How to connect Camel google-pubsub-component to Pubsub emulator?

Question

I have been trying for hours unsuccessfully to get my Spring Boot application's Camel routes (via the Camel component camel-google-pubsub to connect to a local instance of Google Pubsub Emulator.

• My emulator is running :

❯ gcloud beta emulators pubsub start --host-port=localhost:8085 --project=local
Executing: /usr/lib/google-cloud-sdk/platform/pubsub-emulator/bin/cloud-pubsub-emulator --host=localhost --port=8085
[pubsub] This is the Google Pub/Sub fake.
[pubsub] Implementation may be incomplete or differ from the real system.
[pubsub] déc. 01, 2020 6:14:32 PM com.google.cloud.pubsub.testing.v1.Main main
[pubsub] INFO: IAM integration is disabled. IAM policy methods and ACL checks are not supported
[pubsub] déc. 01, 2020 6:14:33 PM com.google.cloud.pubsub.testing.v1.Main main
[pubsub] INFO: Server started, listening on 8085

• My env var is ok : PUBSUB_EMULATOR_HOST=localhost:8085 and I have disabled GOOGLE_APPLICATION_CREDENTIALS in order to avoid a connection to the real PubSub instance

• My Camel route :

@Component
public class MapperInRoute extends ESBRouteBuilder {

    private final String dlFuncTopic;
    private final String subscription;
    private final String routeName;
    private final String getLinksTopic;

    public MapperInRoute(
        final MapperInRouteConfiguration config,
        @Value("${routes.get-links-topic}") final String getLinksTopic
    ) {
        this.dlFuncTopic = config.getDlFuncTopic();
        this.subscription = config.getSubscription();
        this.routeName = config.getName();
        this.getLinksTopic = getLinksTopic;
    }

    @Override
    public void configure() {
        from(format("google-pubsub:{{gcp.pubsub.project-id}}:%s", subscription))
            .routeId(routeName)
            // my route code
            .to(format("google-pubsub:{{gcp.pubsub.project-id}}:%s", getLinksTopic))
        ;
    }
}

• My application-dev.yml :

gcp.pubsub.project-id: local

spring.cloud.gcp.pubsub:
    emulator-host: localhost:8085
    project-id: local

camel.component.google-pubsub.connection-factory.serviceURL: localhost:8085

And finally when I try to run my app locally, my console output is like that :

2020-12-03 12:17:58 INFO   | Starting Application using Java 11.0.2 on lasouche-vm with PID 78394 (/home/mathieu/git/rocher/omnichannel/esb/poc-omega/target/classes started by mathieu in /home/mathieu/git/rocher/omnichannel/esb/poc-omega)
2020-12-03 12:17:58 INFO   | The following profiles are active: dev
2020-12-03 12:17:59 INFO   | Type converters loaded (core: 195, classpath: 10)
2020-12-03 12:17:59 WARN   | UT026010: Buffer pool was not set on WebSocketDeploymentInfo, the default pool will be used
2020-12-03 12:17:59 INFO   | Initializing Spring embedded WebApplicationContext
2020-12-03 12:17:59 INFO   | Root WebApplicationContext: initialization completed in 1696 ms
2020-12-03 12:18:00 INFO   | Registered '/actuator/jolokia' to jolokia-actuator-endpoint
2020-12-03 12:18:00 INFO   | Initialising hawtio services
2020-12-03 12:18:00 INFO   | Configuration will be discovered via system properties
2020-12-03 12:18:00 INFO   | Welcome to Hawtio 2.10.0
2020-12-03 12:18:00 INFO   | Initializing ExecutorService 'applicationTaskExecutor'
2020-12-03 12:18:00 INFO   | Adding welcome page: class path resource [static/index.html]
2020-12-03 12:18:00 INFO   | The default project ID is omega-lab-e5727d
2020-12-03 12:18:00 INFO   | Failed to detect whether we are running on Google Compute Engine.
2020-12-03 12:18:00 WARN   | No core credentials are set. Service-specific credentials (e.g., spring.cloud.gcp.pubsub.credentials.*) should be used if your app uses services that require credentials.
java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:134)
    at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:119)
    at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:91)
    at com.google.api.gax.core.GoogleCredentialsProvider.getCredentials(GoogleCredentialsProvider.java:67)
    at org.springframework.cloud.gcp.core.DefaultCredentialsProvider.<init>(DefaultCredentialsProvider.java:110)
    at org.springframework.cloud.gcp.autoconfigure.core.GcpContextAutoConfiguration.googleCredentials(GcpContextAutoConfiguration.java:61)
    [...]
2020-12-03 12:18:04 INFO   | Initializing ExecutorService 'pubsubPublisherThreadPool'
2020-12-03 12:18:05 INFO   | Initializing ExecutorService 'pubsubSubscriberThreadPool'
2020-12-03 12:18:05 INFO   | Initializing ExecutorService 'pubSubAcknowledgementExecutor'
2020-12-03 12:18:05 INFO   | Exposing 7 endpoint(s) beneath base path '/actuator'
2020-12-03 12:18:06 INFO   | Initialized CamelHttpTransportServlet[name=CamelServlet, contextPath=]
2020-12-03 12:18:06 INFO   | Starting hawtio authentication filter, JAAS authentication disabled
2020-12-03 12:18:06 INFO   | starting server: Undertow - 2.2.2.Final
2020-12-03 12:18:06 INFO   | XNIO version 3.8.0.Final
2020-12-03 12:18:06 INFO   | XNIO NIO Implementation Version 3.8.0.Final
2020-12-03 12:18:06 INFO   | JBoss Threads version 3.1.0.Final
2020-12-03 12:18:06 INFO   | Undertow started on port(s) 8080 (http)
2020-12-03 12:18:06 INFO   | Loading additional Camel XML routes from: classpath:camel/*.xml
2020-12-03 12:18:06 INFO   | Loading additional Camel XML rests from: classpath:camel-rest/*.xml
2020-12-03 12:18:06 INFO   | Apache Camel 2.25.2 (CamelContext: camel-1) is starting
2020-12-03 12:18:06 INFO   | JMX is enabled
2020-12-03 12:18:06 INFO   | StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
2020-12-03 12:18:13 INFO   | Apache Camel 2.25.2 (CamelContext: camel-1) is shutting down
2020-12-03 12:18:13 INFO   | Apache Camel 2.25.2 (CamelContext: camel-1) uptime 7.045 seconds
2020-12-03 12:18:13 INFO   | Apache Camel 2.25.2 (CamelContext: camel-1) is shutdown in 0.080 seconds
2020-12-03 12:18:13 INFO   | stopping server: Undertow - 2.2.2.Final
2020-12-03 12:18:13 INFO   | Destroying hawtio authentication filter
2020-12-03 12:18:13 INFO   | Destroyed CamelHttpTransportServlet[CamelServlet]
2020-12-03 12:18:13 INFO   | Destroying hawtio services
2020-12-03 12:18:13 INFO   | 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-12-03 12:18:13 ERROR  | Application run failed
org.apache.camel.RuntimeCamelException: org.apache.camel.FailedToCreateRouteException: Failed to create route esb-poc-omega-consumer: Route(esb-poc-omega-consumer)[[From[file:/data/loyalty/from/... because of Failed to create Producer for endpoint: google-pubsub://local:POC.OMEGA.CONSUMER.OUT. Reason: java.io.IOException: The Application Default Credentials are not available. They are available if running on Google App Engine, Google Compute Engine, or Google Cloud Shell. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1826)
    at org.apache.camel.spring.SpringCamelContext.start(SpringCamelContext.java:136)
    at org.apache.camel.spring.SpringCamelContext.onApplicationEvent(SpringCamelContext.java:174)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:203)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:196)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:161)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:426)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:383)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:326)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298)
    at com.grouperocher.esb.pocfuse7.Application.main(Application.java:13)
Caused by: org.apache.camel.FailedToCreateRouteException: Failed to create route esb-poc-omega-consumer: Route(esb-poc-omega-consumer)[[From[file:/data/loyalty/from/... because of Failed to create Producer for endpoint: google-pubsub://local:POC.OMEGA.CONSUMER.OUT. Reason: java.io.IOException: The Application Default Credentials are not available. They are available if running on Google App Engine, Google Compute Engine, or Google Cloud Shell. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at org.apache.camel.impl.RouteService.warmUp(RouteService.java:147)
    at org.apache.camel.impl.DefaultCamelContext.doWarmUpRoutes(DefaultCamelContext.java:3954)
    at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:3861)
    at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:3647)
    at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:3488)
    at org.apache.camel.impl.DefaultCamelContext$4.call(DefaultCamelContext.java:3247)
    at org.apache.camel.impl.DefaultCamelContext$4.call(DefaultCamelContext.java:3243)
    at org.apache.camel.impl.DefaultCamelContext.doWithDefinedClassLoader(DefaultCamelContext.java:3266)
    at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:3243)
    at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:72)
    at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:3159)
    at org.apache.camel.spring.SpringCamelContext.start(SpringCamelContext.java:133)
    ... 16 common frames omitted
Caused by: org.apache.camel.FailedToCreateProducerException: Failed to create Producer for endpoint: google-pubsub://local:POC.OMEGA.CONSUMER.OUT. Reason: java.io.IOException: The Application Default Credentials are not available. They are available if running on Google App Engine, Google Compute Engine, or Google Cloud Shell. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at org.apache.camel.impl.ProducerCache.doGetProducer(ProducerCache.java:572)
    [...]
Caused by: java.io.IOException: The Application Default Credentials are not available. They are available if running on Google App Engine, Google Compute Engine, or Google Cloud Shell. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at com.google.api.client.googleapis.auth.oauth2.DefaultCredentialProvider.getDefaultCredential(DefaultCredentialProvider.java:98)
    at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.getApplicationDefault(GoogleCredential.java:217)
    at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.getApplicationDefault(GoogleCredential.java:195)
    at org.apache.camel.component.google.pubsub.GooglePubsubConnectionFactory.createDefault(GooglePubsubConnectionFactory.java:127)
    at org.apache.camel.component.google.pubsub.GooglePubsubConnectionFactory.buildClient(GooglePubsubConnectionFactory.java:101)
    at org.apache.camel.component.google.pubsub.GooglePubsubConnectionFactory.buildClient(GooglePubsubConnectionFactory.java:76)
    at org.apache.camel.component.google.pubsub.GooglePubsubConnectionFactory.getDefaultClient(GooglePubsubConnectionFactory.java:60)
    at org.apache.camel.component.google.pubsub.GooglePubsubEndpoint.afterPropertiesSet(GooglePubsubEndpoint.java:96)
    at org.apache.camel.component.google.pubsub.GooglePubsubEndpoint.createProducer(GooglePubsubEndpoint.java:104)
    at org.apache.camel.impl.ProducerCache.doGetProducer(ProducerCache.java:561)
    ... 118 common frames omitted
2020-12-03 12:18:13 INFO   | Shutting down ExecutorService 'pubSubAcknowledgementExecutor'
2020-12-03 12:18:13 INFO   | Shutting down ExecutorService 'pubsubSubscriberThreadPool'
2020-12-03 12:18:13 INFO   | Shutting down ExecutorService 'pubsubPublisherThreadPool'
2020-12-03 12:18:13 INFO   | Shutting down ExecutorService 'applicationTaskExecutor'
Disconnected from the target VM, address: '127.0.0.1:42485', transport: 'socket'

Process finished with exit code 1

If somebody has already succeeded to use the pubsub emulator with their Camel routes, I am interested in your solution.

Answer 1

I have no clue about Google pub/sub, but the error message says

The Application Default Credentials are not available. They are available if running on Google App Engine, Google Compute Engine, or Google Cloud Shell. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials.

I guess for the simulator you have to define the environment variable GOOGLE_APPLICATION_CREDENTIALS as the error message suggests.

There is also a Google developer page with further details mentioned in the error message.

EDIT

Looking at the Camel-Google-PubSub docs, there is an endpoint-option called endpoint. And it is described as

Endpoint to use with local Pub/Sub emulator.

Have you tried to set your local emulator endpoint as

google-pubsub://project-id:destinationName?endpoint=localhost:8085

Answer 2

I finally managed to connect and use the emulator locally.

First step was to add GOOGLE_APPLICATION_CREDENTIALS env var to my run config (that's not really good because we have to get valid credentials locally in order to connect to Google Pubsub, even if we don't use the real Pubsub).

Then in application-dev.yml :

• I removed spring.cloud.gcp.pubsub and camel.component.google-pubsub.connection-factory.serviceURL properties
• I added the following property : camel.component.google-pubsub.endpoint: localhost:8085 (you were right @burki thank you !)

Finally (and not the least), I upgraded Camel (from 2.23.2 to 3.6.0), I was unable to make my Camel app work with local emulator

PS : Don't forget to create required topics/subscriptions locally before to start you application :

curl -s -X PUT http://localhost:8085/v1/projects/local/topics/my-topic
curl -s -X PUT http://localhost:8085/v1/projects/local/subscriptions/my-subscription -H 'content-type: application/json' -d '{"topic":"projects/local/topics/my-topic"}'
{...}