Running databricks notebooks in ADF: "Invalid client secret provided"

Question

I am currently using a key vault linked service and a databricks linked service in Azure Data Factory to authenticate when running Databricks notebooks. Both linked services are referencing to the same Managed identity object ID. When I try to run my noteook I get the error:

Invalid client secret provided. Ensure the secret being sent in the request is the 
client secret value, not the client secret ID, for a secret added to app 12345-abcd

I've tested the connections of both linked services, and they both pass without any issues.

I've also attempted to follow the method outlined in this resource, which my admin has performed:

https://learn.microsoft.com/en-us/answers/questions/1275176/invalid-client-secret-provided

But I still get the same error.

My admin has also tried to log with az login with the secret id as service principal and it works.

Are we missing something when Azure Data Factory? Is it possible to see where client secret and id are referenced?

Answer 1

Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app

I tried to reproduce your issue and got the same error, and the reason is while you try to authenticate blob storage with service principal you need to pass the correct values for the Client Id, Tenant Id and Client Secret Or you will face this error.

The issue is client secret you are using to authenticate is incorrect

To resolve this check:

• Verify that the secret is not expired.
• Ensure that the client secret you are using is correct unmodified or corrupted during copy-pasting.
• Check if the client secret is being referenced correctly in all of your config settings.

To access values from key vault in databricks you can create scope for key vault at databricks and access secret values in code with below command.

CLIENT_SECRET=dbutils.secrets.get(scope="sampleakv1", key="clientsecret")

if you are passing it from data factory parameters or variables use below command

dbutils.widgets.get("parameter name/variable name")