ArgoCD Callback To Pipeline

Question

What I want is to have ArgoCD approve a job that is in the approval step in CircleCI when a deployment was successful.

However, this has been a multi week journey with no results. I first looked at this:

https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/

I spent the time building out an entire argo workflow, and job. Then setup a pipeline to capture the waiting job id (before updating the yaml file) and run the workflow with that id that would generate a PostSync that could use that id to approve the waiting circleci job only AFTER a successful deploy, then destroy itself. I also had one for SyncFail as well. My hopes were dashed when I realized this was generic. I couldn't see a way to only run this job for a specific application in argocd, it appears to run on ANY sync event. (What??)

Now I'm looking at this: https://argo-cd.readthedocs.io/en/stable/operator-manual/notifications/

However, yet again I see hiccups already before I event started. There seems no possible way to run custom code, therefore I can't run the API to approve my job. Has anyone gotten a callback system for a specific app working? I can try and hack something together by calling back to a pipeline to run an API, but that just seems dumb. There has to be a better way....

Answer 1

Nearly a month of searching, building, and digging through code later and I have something working. Wasn't exactly what I wanted above but I think this is probably better in the long run anyway.

Instead of a callback mechanism I went with a async callback, although I will give you some ideas of ways you can implement the callback approval mechanism below.

Background

I'm using the following services:

    argocd
    argocd-notifications
    argocd-workflows

By default argocd-notifications comes bundled with argocd while argocd-workflows does not and is required to have a separate install. (all done via the community helm charts).

ArgoCD-Notifications Setup

argocd-notifications-cm (configmap MUST be named this, argocd will auto pickup this configmap name) argocd-notifications-secret (secret MUST be named this for the same reason)

argocd-notifications-cm

Please note, if you just use this as is you will overwrite the one that is generated by default. You can append this to the existing map if you want, just be sure to capture that information BEFORE applying your own.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  template.run-github-pipeline: |
    webhook:
      trigger-workflow:
        method: POST
        path: /submit
        body: |
          {
            "resourceKind": "WorkflowTemplate",
            "namespace": "argo-workflows",
            "resourceName": "scripts-trigger-pipelines",
            "submitOptions": {
                "entryPoint": "github",
                "generateName": "{{ `{{.app.metadata.name}}` }}-github-pipeline-",
                "parameters": [
                    "app_name={{ `{{.app.metadata.name}}` }}",
                    "trigger_with_data={{ `{{.recipient}}` }}"
                ],
                "labels": "workflows.argoproj.io/workflow-template=scripts-trigger-pipelines",
                "generateName": "{{ `{{.app.metadata.name}}` }}-trigger-github-pipeline-"
          }}
  trigger.on-deployment-success-circleci: |
    - when: app.status.sync.status in ['Synced'] and app.status.operationState != nil and app.status.operationState.phase in ['Succeeded'] and app.status.health.status in ['Healthy']
      oncePer: app.status.operationState.syncResult.revision
      send: [run-circleci-pipeline]
  service.webhook.trigger-workflow: |
    url: "https://{{ .Values.workflows.endpoint }}/api/v1/workflows/argo-workflows"
    headers:
    - name: "Authorization"
      value: "Bearer $ARGO_WORKFLOWS_API_TOKEN"
    - name: "Content-Type"
      value: "application/json"
    insecureSkipVerify: false

argocd-notifications-secret

This secret must also exist so that it will know what the $ARGO_WORKFLOWS_API_TOKEN is:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets-manager
  refreshInterval: 1m
  target:
    creationPolicy: Owner
    deletionPolicy: Delete
  # map secret key/value pairs to envvars
  data:
    - secretKey: ARGO_WORKFLOWS_API_TOKEN
      remoteRef:
        key: my-secret-name
        property: argo_workflows_api_token

In the above I'm using an external secret so I can maintain the secret elsewhere and have it synced to kuberentes but you can also just use a plain Secret if you want. Subscription

Now this one got me for a while, but you can ONLY add these annotations to the kubernetes resource kind: Application. This is something that only argocd understands. You can add it like:

notifications.argoproj.io/subscribe.on-deployment-success-github.trigger-workflow: 'repo:my_test_repo|eventtype:my-repository-dispatch-event|payload:{app_name:sample}'

This takes advantage of the {{.recipient}} as defined in the configmap above. It will add the following into the {{.recipient}} entry on that configmap BEFORE firing the event:

repo:my_test_repo|eventtype:my-repository-dispatch-event|payload:{app_name:sample}

Now the argo-notifications side is completed. We need to setup the argo-workflows side....

Argo-Workflows Setup

You will need to setup some way to communicate with argo-workflows. That means you will need to setup a webhook (sorry won't go into detail how to do that here, just know you need to have it). Get API Token

Next we need to setup a service account, rolebinding, and role that will we use to access the api of argo-workflows. In order to get the api token that we use in the configmap above for notifications we need to generate a secret:

apiVersion: v1
kind: Secret
metadata:
  name: argo-workflows-pipelines.service-account-token
  annotations:
    kubernetes.io/service-account.name: argo-workflows-pipelines
  type: kubernetes.io/service-account-token

Doing this will then allow us to run the following command to get the api token and place it either into the external secret or the argocd-notifications-secret directly:

kg secret argo-workflows-pipelines.service-account-token -o=jsonpath='{.data.token}' -n argo-workflows | base64 --decode

Setup Job Manifest

Now that we have the token we can setup a job manifest that will be triggered by the noficiation which will be in charge of calling back to any pipeline we want (circleci, github, bitbucket, etc.) -- in this example I'm using github:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: scripts-trigger-pipelines
spec:
  ttlStrategy:
    secondsAfterCompletion: 320    # Auto delete workflow in 5 minutes after it completes
  entrypoint: github
  arguments:
    parameters:
      - name: trigger_with_data
        value: "{}"
      - name: app_name
        value: ""
  templates:
    - name: github
      serviceAccountName: <my-sa-name>
      inputs:
        parameters:
          - name: trigger_with_data
          - name: app_name
      metadata:
        annotations:
          kubectl.kubernetes.io/default-container: main
      resource:
        action: create
        setOwnerReference: true
        manifest: |
          apiVersion: batch/v1
          kind: Job
          metadata:
            generateName: "{{ `{{inputs.parameters.app_name}}` }}-trigger-github-pipeline-"
          spec:
            template:
              spec:
                serviceAccountName: argo-workflows-pipelines
                volumes:
                - name: secret-volume
                  secret:
                    secretName: <my-secret-volume>
                containers:
                - name: trigger-github-pipeline
                  image: {{ .Values.awsAccountID }}.dkr.ecr.us-east-1.amazonaws.com/{{ .Values.pipelineCallbacks.docker.image }}:{{ .Values.pipelineCallbacks.docker.tag }}
                  command: ["/bin/bash", "-c"]
                  args:
                    - |-
                      echo "Input App: {{ `{{inputs.parameters.app_name}}` }}"
                      INPUT_DATA='{{ `{{inputs.parameters.trigger_with_data}}` }}'
                      ...


                      # Trigger target github repository dispatch event with the generated token and parsed data
                      echo "Issuing curl to url:https://api.github.com/repos/<owner>/$REPO/dispatches -- with data: {\"event_type\":\"$EVENT_TYPE\",\"client_payload\":{\"dispatch_payload\":$PAYLOAD}}"
                      curl \
                        -X POST \
                        -H "Accept: application/vnd.github+json" \
                        -H "Authorization: Bearer $GITHUB_TOKEN"\
                        -H "X-GitHub-Api-Version: 2022-11-28" \
                        https://api.github.com/repos/<owner>/$REPO/dispatches \
                        -d "{\
                          \"event_type\":\"$EVENT_TYPE\",\
                          \"client_payload\":$PAYLOAD
                        }"

                  volumeMounts:
                  - name: secret-volume
                    readOnly: true
                    mountPath: "/secrets"
                restartPolicy: Never
            backoffLimit: 2

Note: I don't spell everything out for you here, as you will need to get your GITHUB_TOKEN into the workflow. I would just use my provided example for "secrets" via a volume mount. I would use a github app to dynamically generate the token as well, more secure.

NOTE2:

ttlStrategy:
    secondsAfterCompletion: 320

While first building I would probably remove this until you're ready for it. Github Setup

Now that we have argo-notifications and argo-workflows setup we can setup a repository dispatch event inside our github pipeline (or something equivalent for other pipelines)

name: "Argo Callback"
on:
  repository_dispatch:
    types: [my-repository-dispatch-event]
    inputs:
      app_name:
        description: 'The name of the app you want to update in your own triggering repo.'
        required: true
        type: string

jobs:
  argo-callback:
    name: "Init Callback"
    runs-on: ubuntu-latest
    steps:
      - name: Output payload
        id: verify
        run: |
          echo "RECEIVED CALLBACK FOR APP: ${{ github.event.client_payload.app_name }}"
          echo "${{ github.event.client_payload }}"

As long as I didn't typo something and you filled everything out it should be working.

However, what if it isn't? Lets talk about debugging. Debugging

Argo-Notifications

kubectl get pods -n argocd | grep argocd-notifications

Should spit out the notifications controller. Then you can look at it's logs:

kubectl logs <pod_name> -n argocd --tail=100 -f

I added -f if you want to follow along the logs in realtime.

Argo-Workflows

kubectl get pods -n argo-workflows

there is a server and a controller just like argo-notifications. Look at those logs using the same method. That should give you all the information you need to succeed! Good luck!

Argo-Notifications documentation was hell. Also here is some helpful info on un-documented things like app.status etc.

https://github.com/argoproj/gitops-engine/blob/187312fe86ed627b1cf4147c13dccf9ff331424b/pkg/sync/sync_task.go

https://github.com/argoproj/gitops-engine/blob/master/pkg/sync/common/types.go#L54

Callback Approval

Now with all the above you can achieve a callback approval if you use a database, something like dynamo db to store the hash that is triggering the deployment. Then use the undocument app.status.operationState.syncResult.revision (Which is basically the short hash for the commit) parameter inside your configmap to send it to argo-workflows. Then argo-workflows will use that hash to lookup that information in the database to know who it should respond to.

I know this is going to be a super valuable resource for anyone working with workflows or notifications.