So, here is the link to Jobs API call in Databricks here
Everything works in Python using requests. E.g both the job creation and job listing works
import requests
proxy= "http://127.0.0.1:8888"
access_token="tokenabc"
proxies = {
"https": proxy,
}
header_read = {
'Authorization': "Bearer " + access_token,
'Accept': "application/scim+json"
}
#list jobs
init_get=requests.get('https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/list', headers=header_read, proxies=proxies)
#create job
init_post=requests.post('https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/create', headers=header_read, proxies=proxies,verify=False,json=job_payload)
However, strangely in Powershell, only the job creation works and Job listing fails.
#this works
Invoke-RestMethod -Uri 'https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/create' -Headers @{ 'Authorization' = "Bearer $bearertoken" } -Method Post -Body $content -ContentType 'application/json'
#this does not work
Invoke-WebRequest -Uri 'https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/list' -Headers @{ 'Authorization' = "Bearer $bearertoken" } -Method Get
#RestMethod also does not work
Invoke-RestMethod -Uri 'https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/list' -Headers @{ 'Authorization' = "Bearer $bearertoken" } -Method Get
I have also tried setting the Content type on these but nothing helps. Also, explicitly setting the proxy (fiddler) does not help.
-proxy "http://127.0.0.1:8888"
But should not be the proxy also , as the post method works.
I just keep getting an error like
Invoke-WebRequest : The underlying connection was closed: An unexpected error occurred on a send.
At line:22 char:5
+ Invoke-WebRequest -Uri 'https://databricksworkspaceid.azuredatabricks.net/ap ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
or in case of RestMethod
Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send.
At line:22 char:5
+ Invoke-RestMethod -Uri 'https://databricksworkspaceid.azuredatabricks.net/ap ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
I could have understood if everything failed in Powershell, but the post method (job creation) works, so not sure, why the connection would be terminated in case of a Get request but not a post.
Going by some forum posts, I have also tried the following but to no avail-
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Does anyone know what am I doing wrong/missing? Baffling to see working in python but only a part in Powershell.
So, I finally found the answer. Basically 2 things:
Basically allowing TLS, TLS1.1 and TLS1.2