Encountering "[Errno 16] Device or Resource Busy" Error with Kafka Consumer in Snowflake Stored Procedure

Question

Goal

The goal is to create a Snowflake stored procedure that consumes messages from a Kafka topic once per day, processes these messages, and then loads the processed data into Snowflake tables for further analysis.

Error Description:

The error encountered is [Errno 16] Device or resource busy. It occurs specifically at the line messages = consumer.poll(timeout_ms=1000) in the Kafka consumer code. This issue only arises when running the code within Snowflake; the same code executes successfully in my local environment.

Code Snippet:

from kafka import KafkaConsumer
 
# Kafka consumer configuration
consumer_config = {
    'bootstrap_servers': 'XXX',
    'security_protocol': 'SASL_SSL',
    'sasl_mechanism': 'PLAIN',
    'sasl_plain_username': 'XXX',
    'sasl_plain_password': "XXX",
    'client_id': 'snowflake-dev-client',
    'group_id': 'ZZZ',
    'auto_offset_reset': 'earliest',
    'enable_auto_commit': True
}
 
# Initialize Kafka consumer
consumer = KafkaConsumer(
    "all",
    **consumer_config
)
 
message_list = []
MAX_EMPTY_POLLS = 5  # Maximum number of empty polls before breaking the loop
empty_polls = 0
 
try:
    while empty_polls < MAX_EMPTY_POLLS:
       # Poll for messages
        messages = consumer.poll(timeout_ms=1000)  # Timeout in milliseconds
        if not messages:
            empty_polls += 1
            time.sleep(1)  # Optional: sleep to avoid tight looping
            continue
 
        empty_polls = 0  # Reset empty poll count if messages are found
 
        for tp, msgs in messages.items():
            for message in msgs:
                # Process each message
                msg_decoded = message.value.decode('utf-8')
                msg_json = json.loads(msg_decoded)
                if 'YYY' in msg_json['event_name']:
                    message_list.append(add_values_to_list(msg_json))
 
except Exception as e:
    print(f"Error: {e}")
finally:
    # Close the consumer
    consumer.close()

Possible errors that were checked

• Network Rules: The network rules associated with external access integration have been verified. The connection to the Kafka bootstrap server is successful, indicating proper network setup.
• Authentication Keys: The keys used for Kafka authentication have also been checked. The successful connection confirms their validity.

Questions:

1. Are there known limitations or special configurations required for external connections (like Kafka) in Snowflake stored procedures?
2. Has anyone successfully used Kafka consumers in Snowflake stored procedures, and if so, how did you manage connection conflicts or library compatibility?
3. Any insights into why this error might be occurring specifically in the Snowflake environment?