Google PubSub Subscriber issues with Google SDM API (Nest Devices)

Question

So, I'm stuck. This may be lengthy as I want to establish all (or most of what I did) to get to this point:

Background: Want to create an app for the Raspberry PI which can access and act upon the Nest suite of devices (Doorbell, Thermostat, etc.). Am currently doing this in an Ubuntu VM on a laptop.

What I have done:

1. Created a Device Access project, following the steps which google provides.
2. Also created a GCP Project (not the same thing as above), so as to be able to set up the OAuth and Subscription settings
3. The Device Access project uses the OAuth from the GCP project, i.e. the Device Access project has a field where the GCP project OAuth is referenced
4. COmpleted the Device Access project steps, granting permissions through browser pages, up to the point where the authorization code is created and used by me, and I get my first access token. I am able to execute CURL commands to retrieve Nest device characteristics and even a RTSP url to get access to the Camera stream.

So I am half of the way there, in that I can interact with the devices (REST API) by the use of OAuth access tokens. But what I want to do is be able to react to events from the devices, which requires PubSub interactivity. This is where I am stuck

Extra Step: I ALSO created a service account in GCP Credentials, because I had seen in the documentation that it may be required, and googles algorithm requires a GOOGLE_APPLICATION_CREDENTIALS environment variable to a json file with the downloaded credentials of the service account. So check

5. While setting up the Device Access project, a PubSub topic was created. In the GCP console, I created a corresponding Subscription for the GCP project and referenced it manually to the topic created as part to the Device Access project (steps above)
6. It is mentioned in the documentation that the first time you interact via REST API with the OAuth access token, the pubsub functionality becomes available. NB: I can confirm this "Viewing Messages" from the PubSub Subscription pages in GCP Console. I am able to "pull" messages and see the messages appear in the subscription. I AM aware that by pulling messages, I am temporarily preventing them from appearing in other queues, but I wanted to check that the messages are able to be populated in the subscription.

Ok. So, no I want to write the code to access the pubsub subscription and create my app (or part of it) around reacting to these event meessages. This is where I have been stuck for two days.

I have created a python script based on the method provide by google (I don't mind including IDs since this is not private)

from google.cloud import pubsub_v1
from concurrent.futures import TimeoutError

project_id = 'nestcontroller-299520'
subscription_id = 'NestEventPull'
timeout = 60.0

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)

def callback(message):
    print(f"Received a message: \n")
    print(f"{message} \n")
    message.ack()

future = subscriber.subscribe(subscription_path, callback=callback)
print(f"\nprojtest - listening for messages on: {subscription_path}\n")
print(f"subscription_id : {subscription_id}\n")

with subscriber:
    try:
        future.result(timeout=timeout)
    except:
        TimeoutError
        future.cancel()

The output was the following:

projtest - listening for messages on: projects/nestcontroller-299520/subscriptions/NestControlSub

subscription_id : NestControlSub

Background thread did not exit.
ERROR:root:Exception in callback <bound method ResumableBidiRpc._on_call_done of <google.api_core.bidi.ResumableBidiRpc object at 0x7f042ae47fa0>>: ValueError('Cannot invoke RPC: Channel closed!')

The code above got "stuck" for 60 seconds, after which the "Background thread did not exit." and ERROR appeared

I tried several variations of the above, including another python script using (i.e. "PIP INSTALL"ing) google-cloud-pubsub libraries, but ultimately I tried the following, which I borrowed from another solution I found in an attempt to find out what was going on while it was "stuck".

    project_id = 'nestcontroller-299520'
    subscription_id = 'NestControlSub'
    topic_id = 'projects/sdm-prod/topics/enterprise-5a0f7c4e-d20d-4e56-968d-c6239936a3b0'


    def callback(message, event):
        logger.info(message)
        event.set()

    subscriber = pubsub_v1.SubscriberClient()

    event = Event()

    def callback_wrapper(message):
        callback(message, event)

    future = subscriber.subscribe('subscription/path', callback=callback_wrapper)

    event.wait()
    logger.exception('Got event. Shutting down.')
    future.cancel()
    exit(1) 

which immediately resulted in the following "Invalid grant: account not found" error, which kept repeating over and over until I cancelled the script...

ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7f7188d17d30>" raised exception!
Traceback (most recent call last):
  File "/home/christian/PythonProjects/google-cloud-pubsub/venv/lib/python3.8/site-packages/grpc/_plugin_wrapping.py", line 77, in __call__
    self._metadata_plugin(
  File "/home/christian/PythonProjects/google-cloud-pubsub/venv/lib/python3.8/site-packages/google/auth/transport/grpc.py", line 86, in __call__
    callback(self._get_authorization_headers(context), None)
  File "/home/christian/PythonProjects/google-cloud-pubsub/venv/lib/python3.8/site-packages/google/auth/transport/grpc.py", line 72, in _get_authorization_headers
    self._credentials.before_request(
  File "/home/christian/PythonProjects/google-cloud-pubsub/venv/lib/python3.8/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/home/christian/PythonProjects/google-cloud-pubsub/venv/lib/python3.8/site-packages/google/oauth2/service_account.py", line 361, in refresh
    access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion)
  File "/home/christian/PythonProjects/google-cloud-pubsub/venv/lib/python3.8/site-packages/google/oauth2/_client.py", line 153, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/home/christian/PythonProjects/google-cloud-pubsub/venv/lib/python3.8/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/home/christian/PythonProjects/google-cloud-pubsub/venv/lib/python3.8/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid grant: account not found', '{"error":"invalid_grant","error_description":"Invalid grant: account not found"}')
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed non-terminating stream error 503 Getting metadata from plugin failed with error: ('invalid_grant: Invalid grant: account not found', '{"error":"invalid_grant","error_description":"Invalid grant: account not found"}')
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 503 Getting metadata from plugin failed with error: ('invalid_grant: Invalid grant: account not found', '{"error":"invalid_grant","error_description":"Invalid grant: account not found"}')
INFO:google.api_core.bidi:Re-established stream
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed non-terminating stream error 503 Getting metadata from plugin failed with error: ('invalid_grant: Invalid grant: account not found', '{"error":"invalid_grant","error_description":"Invalid grant: account not found"}')
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 503 Getting metadata from plugin failed with error: ('invalid_grant: Invalid grant: account not found', '{"error":"invalid_grant","error_description":"Invalid grant: account not found"}')

Does anyone have any idea why the Pubsub model is not working, even though I have followed all possible documentation variants? I have tried changing the credential file details (it gives clear defined errors whenever I DON'T use the Service Account credentials, so that does not seem to be the issue.

Any help or hints will be greatly appreciated.

Answer 1

Ok. So after a good night's sleep and a lucky YouTube video, I figured out the missing step.

The Role(s) of the Service Account have to be specifically configured for the Pub/Sub Publisher and Pub/Sub Subscriber roles. Once this was done and the Service Account re-activated, the evnet messages started coming through

Anyway, I hope the process above and the conclusion helps anyone stuck with the same problem