Benthos pipleline to read XML from postman and publish to kafka topic

32 Views Asked by At

i want to create a pipeline to read XML data from postman http url and consume it through benthos input configuration and publish this message to a kafka topic using benthos processor . Following is the configuration , i was trying but doesn't seem working

input:
  http_server:      
      address: ""      
      path: /
      ws_path: /ws
      allowed_verbs:        
        - POST
      timeout: 5s
      rate_limit: ""
pipeline:
  processors:
    - xml: {}
output:
  kafka:
    addresses:
      - abc-central-1-kafka-2-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
      - abc-central-1-kafka-1-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
      - abc-central-1-kafka-0-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
    topic: abc_abc-central-1.abc-tyur-service-in.v1
    client_id: abc-tyur-service-dev-1
    tls:
      enabled: true
      root_cas_file: /Users/ca.crt
      client_certs:
        - cert_file: /Users/cert.pem
          key_file: /Users/key.pem

logger:
  level: ALL
1

There are 1 best solutions below

0
lucky On BEST ANSWER

When we don't want to modify our input messages and want to publish unmodified messages to the output , processor can be removed and only input and output configuration are sufficient. Removing processor solved my problem . Following the working configuration

input:
  http_server:      
      address: ""      
      path: /
      ws_path: /ws
      allowed_verbs:        
        - POST
      timeout: 5s
      rate_limit: ""
output:
  kafka:
    addresses:
      - abc-central-1-kafka-2-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
      - abc-central-1-kafka-1-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
      - abc-central-1-kafka-0-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
    topic: abc_abc-central-1.abc-tyur-service-in.v1
    client_id: abc-tyur-service-dev-1
    tls:
      enabled: true
      root_cas_file: /Users/ca.crt
      client_certs:
        - cert_file: /Users/cert.pem
          key_file: /Users/key.pem

logger:
  level: ALL

Thanks to benthos community on Discord