Argo Events: Exposing Webhook Through a K8s Load Balancer on Azure Subnet

Question

I'm trying to route POST requests through a K8s Load Balancer to a Webhook in Argo Events. I can't find any clear documentation on this. I'm able to get the Webhook created and I can successfully communicate with it when I port forward the webhook-eventsource-svc. The Load Balancer is built fine and displays the external IP that I assign. However when I try to POST to the Load Balancer I just get a connection timed out error. I'm hoping I'm just configuring these manifests wrong.

Here is the manifest for both services.

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: webhook
  namespace: argo-events
spec:
  service:
    ports:
    - port: 12000
      targetPort: 12000
  webhook: 
    example:
      endpoint: /deploy
      method: POST
      port: "12000"
---
apiVersion: v1
kind: Service
metadata:
  name: webhook-loadbalancer
  namespace: argo-events
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: DevelopSubnet
spec:
  type: LoadBalancer
  loadBalancerIP: 1XX.X.X.XXX
  ports:
  - protocol: TCP   
    port: 90
    targetPort: 12000
  selector:
    app: webhook-eventsource-svc
    controller: eventsource-controller

And here is how I am sending the request:

curl -d '@params.json' -H "Content-Type: application/json" -X POST http://1XX.X.X.XXX:90/deploy

Any suggestions?

Answer 1

just wanted to share how you expose the webhook so depending on the documentations when you want to expose a webhook

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: webhook
spec:
  # to expose this endpoint to public internet we need to remove this part
  # service:
  #   ports:
  #     - port: 12000
  #       targetPort: 12000
  webhook:
    example:
      port: "12000"
      # now this is the endpoint you may want to expose with your ingress/virtualService/etc..
      endpoint: /webhook
      method: POST

the next part is to create a service that will point to the webhook

apiVersion: v1
kind: Service
metadata:
  name: webhook-eventsource
spec:
  ports:
    - port: 12000
      protocol: TCP
      name: tcp
      targetPort: 12000
  selector:
    # pods created from `EventSource` will have the label of `eventsource-name=eventsourcename` by default so we use this selector to select it
    eventsource-name: webhook
  type: ClusterIP

last but not least you will have to expose the service with an endpoint (i am using ISTIO in my example here with virtual service)

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: https
  namespace: istio-system
spec:
  hosts:
    - "*"
  gateways:
    - http-gateway
  http:
    # this URL matching should match the URL inside the following "EventSource.webhook.example"
    # it is not intended to be used like "/webhook/webhook" the URL should directly match the one inside the object "webhook"
    - match:
        - uri:
            prefix: /webhook
      route:
        - destination:
            host: webhook-eventsource.argo-events.svc.cluster.local
            port:
              number: 12000

it is very important to notice here that both /webhook inside the EventSource is exactly matching the URL in your ingress/virtualService

Answer 2

I'm trying to do something similar in AWS. I can get the sample webhook to work with port forwarding (https://argoproj.github.io/argo-events/quick_start/) But it won't work with regular K8s objects. In my case, an Ingress and a Service object. I can see my Service selector correctly pick the webhook sensor pod. Both Argo Events and Argo Workflow run in the same argo namespace. Once configured, access to the Ingress from Postman returns a 404. What I find confusing is that the actual Port the sensor pod exposes is 7777 in the sample, not 12000. So, I've tried a Service with Port 12000 / TargetPort 12000 or 7777. In either case, the POST returns 404.

What I can point out that's applicable in your case and mine is this (https://argoproj.github.io/argo-events/eventsources/services/) in the second paragraph it states that you must remove the service field from your EventSource object to refactor the sample from port forwarding. Hope it helps. I'm still trying to make this work.