Kafka producer producing message but consumer not consuming using Python

88 Views Asked by At

I am working with AWS EC2 instance. I am able to produce and consume data normally with kraft in terminal from my ubuntu 22.04 using ssh. When I try to produce data using a python script to the same topic - topic name "demo_test", i am able to produce it but nothing is being consumed by the corresponding consumer script or the terminal.

When i try to produce to and consume from a topic in terminal, it is wrking fine but when i try to achieve the same thing using jupyter, it is not working.

I have created an EC2 instance on AWS and have installed java and kafka 2.12-3.3.6 on it and am connecting to the instance using ssh ubuntu terminal on virtualbox

I have written the following code using kafka-python in jupyter:

Producer code:-

import pandas as pd
from kafka import KafkaConsumer, KafkaProducer
from time import sleep
from json import dumps
import json

producer = KafkaProducer(
 bootstrap_servers=['50.19.162.182:9092'],
 value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

print(producer)
# prints -> <kafka.producer.kafka.KafkaProducer object at 0x7f2e4c21a200>

producer.send('demo_test', value='{"hello":"World"}')
# prints -> <kafka.producer.future.FutureRecordMetadata at 0x7f2e4c7f05b0> as output

Consumer code:-

from kafka import KafkaConsumer
from time import sleep
from json import loads
import json

consumer = KafkaConsumer('demo_test',
 bootstrap_servers=['50.19.162.182:9092'],
 value_deserializer=lambda v: json.loads(v.decode('utf-8'))
)

print(consumer)
# prints -> <kafka.consumer.group.KafkaConsumer object at 0x7faad50af160>

for c in consumer:
  print(c.value)

I am new to kafka so any help is highly appreciated.

0

There are 0 best solutions below