I'm trying to automate creation of connector in airflow by github action, but since it is an external provider, the payload that need to be sent to airflow REST API doesn't work and i didn't find any documentation on how to do it.
So here is the PAYLOAD i'm trying to send :
PAYLOAD = {
"connection_id": CONNECTOR,
"conn_type": "google_cloud_platform",
"extra": json.dumps({
"google_cloud_platform": {
"keyfile_dict" : open(CONNECTOR_SERVICE_ACCOUNT_FILE, "r").read(),
"num_retries" : 2,
}
})
}
According to the airflow documentation here
And the information i found on the "create connector" page of airflow UI : Airflow UI create connector page
But i received no error (code 200) and the connector is created but doesn't have the settings i tried to configure.
I confirm the creation works on the UI.
Does anyone have a solution or document that refer to the exact right payload i need to sent to airflow rest api ? Or maybe i miss something.
- Airflow version : 2.2.3+composer
- Cloud Composer version (GCP) : 2.0.3
- Github runner version : 2.288.1
- Language : Python
Thanks guys and feel free to contact me for further questions.
Bye
You might be running into encoding/decoding issues while sending data over the web.
Since you're using Composer, it might be a good idea to use Composer CLI to create a connection.
Here's how to run airflow commands in Composer:
Here's how to create a connection with the native Airflow commands:
Combining the two, you'll get something like:
You could run this in a Docker image where
gcloud
is already installed.