I am working with AWS EC2 instance. I am able to produce and consume data normally with kraft in terminal from my ubuntu 22.04 using ssh. When I try to produce data using a python script to the same topic - topic name "demo_test", i am able to produce it but nothing is being consumed by the corresponding consumer script or the terminal.
When i try to produce to and consume from a topic in terminal, it is wrking fine but when i try to achieve the same thing using jupyter, it is not working.
I have created an EC2 instance on AWS and have installed java and kafka 2.12-3.3.6 on it and am connecting to the instance using ssh ubuntu terminal on virtualbox
I have written the following code using kafka-python in jupyter:
Producer code:-
import pandas as pd
from kafka import KafkaConsumer, KafkaProducer
from time import sleep
from json import dumps
import json
producer = KafkaProducer(
bootstrap_servers=['50.19.162.182:9092'],
value_serializer=lambda v: json.dumps(v).encode('utf-8')
)
print(producer)
# prints -> <kafka.producer.kafka.KafkaProducer object at 0x7f2e4c21a200>
producer.send('demo_test', value='{"hello":"World"}')
# prints -> <kafka.producer.future.FutureRecordMetadata at 0x7f2e4c7f05b0> as output
Consumer code:-
from kafka import KafkaConsumer
from time import sleep
from json import loads
import json
consumer = KafkaConsumer('demo_test',
bootstrap_servers=['50.19.162.182:9092'],
value_deserializer=lambda v: json.loads(v.decode('utf-8'))
)
print(consumer)
# prints -> <kafka.consumer.group.KafkaConsumer object at 0x7faad50af160>
for c in consumer:
print(c.value)
I am new to kafka so any help is highly appreciated.