GoogleCredentials.getApplicationDefault() returning unexpected results

Question

I am using a Google cloud function and can't get the default service account to work properly. The service account works fine if I pass in a JSON credentials file. However, if I set the same service account to be the default account for the function I get an error. Also, the code works fine if I run locally and use the defalt service account which is also set up to the same account. The error I get when the function runs in the cloud is:

The user does not belong to a G Suite customer.

This generally means either the service account is not authorized for domain wide delegation or that no G-Suite user was specified for impersonation. Usage metrics from the cloud console indicate the correct service account is used to run the function. Also this account appears in the run configuration section of the function as expected. I am using both G-suite functions(Vault) and Google cloud functions (storage). The service account is set up for domain wide delegation and I am setting a Gsuite user to impersonate.

In summary, the same service account and user:

• Works in the cloud or desktop if I pass in a JSON credentials file,
• Does not work in the cloud with the same service account being set as the default,
• Works on my laptop with the same service account set as the Default Service Account.

The run tab on the function show the correct service account and the usage statistics also indicate the correct service account is being used. I want to use the Default Service Account so I don't have to manage storing and retrieving credentials.

The relevant code is as follows:

public HttpRequestInitializer buildInitializer() throws Exception {
    if (VaultConfig.isApplicationRunMode())
        return new HttpCredentialsAdapter(buildDefaultCredentials());
    else if (VaultConfig.isServiceRunMode())                    
        return new HttpCredentialsAdapter(buildServiceCredentials());
    else throw new Exception("Request initializer only supports service account or application default");
}

private GoogleCredentials buildServiceCredentials() throws Exception {
    String credentialsPath = VaultConfig.getServiceCredentials();
    logger.info("Credentials path " + credentialsPath);
    InputStream is=GetRequestInitializer.class.getResourceAsStream(credentialsPath);
    if (is == null) {
        throw new FileNotFoundException("Resource not found: " + credentialsPath);
    }
    GoogleCredentials credentials = GoogleCredentials.fromStream(is)
            .createScoped(VaultConfig.getScopes()).createDelegated(VaultConfig.getUserToImpersonate());
    logger.debug("Credentials created.  Scopes " + VaultConfig.getScopes());
    logger.debug(credentials.toString());
    return credentials;
}

private GoogleCredentials buildDefaultCredentials() throws Exception {
    logger.info("Using application default credentials");
            GoogleCredentials credentials = GoogleCredentials.getApplicationDefault()
            .createScoped(VaultConfig.getScopes())
            .createDelegated(VaultConfig.getUserToImpersonate());
    logger.debug("Credentials created.  Scopes " + VaultConfig.getScopes());
    logger.debug(credentials.toString());
    return credentials;
}

The error occurs later in the code where I actually call the Vault service and the server returns a 400 response with the indicated error. Please note the debug statement logger.debug(credentials.toString()). In all situations where the function works the credentials.toString() returns expected results as follows:

ServiceAccountCredentials{clientId=1111111111222233, [email protected], privateKeyId=aaaabb12bgbaaaaaaaaaa, transportFactoryClassName=com.google.auth.oauth2.OAuth2Utils$DefaultHttpTransportFactory, tokenServerUri=https://oauth2.googleapis.com/token, scopes=[https://www.googleapis.com/auth/devstorage.read_write, https://www.googleapis.com/auth/devstorage.full_control, https://www.googleapis.com/auth/ediscovery], defaultScopes=[], [email protected], quotaProjectId=null, lifetime=3600, useJwtAccessWithScope=false}

However, when using the Default service account in the cloud credentials.toString() returns

ComputeEngineCredentials{transportFactoryClassName=com.google.auth.oauth2.OAuth2Utils$DefaultHttpTransportFactory}

It looks to me like my problem is that GoogleCredentials.getApplicationDefault() does not return the expected result when run on the cloud. I set up the service account following https://cloud.google.com/docs/authentication/production.

Is there something else I need to do? Any tips on additional diagnostic statements that might verify that the problem is? The run tab on the function shows the correct account and I am using what should be the default account for the project.