Why is my Python asyncio script returning None when fetching non-empty data from Redis?

Question

I'm encountering an issue in my Python script where I asynchronously fetch and save data from multiple exchanges into Redis. The goal is to periodically update the data and retain only the most recent records in Redis. Despite the log showing successful data saving, and the data not being empty, the fetching function persistently returns None.

Saving Data to Redis:

import asyncio
import logging
from redis.asyncio.client import Redis
import json

logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s', level=logging.INFO)

async def fetch_and_save(redis_pool, exchange, method, symbol=None, interval=1):
    while True:
        try:
            data = await getattr(exchange, method)() if symbol is None else await getattr(exchange, method)(symbol)
            key = f"{exchange.__class__.__name__}:{method}"
            serialized_data = json.dumps(data)
            if symbol:
                key = f"{key}:{symbol}"
            await redis_pool.set(key, serialized_data)
            logging.info(f"Data saved for {key}")
        except Exception as e:
            logging.error(f"Error fetching {method} from {exchange.__class__.__name__}: {e}")
        await asyncio.sleep(interval)

Fetching Data from Redis:

async def fetch_data_from_redis(redis_host, exchange_name, method, symbol=None):
    redis = Redis(host=redis_host)
    key = f"{exchange_name}:{method}"
    if symbol:
        key = f"{key}:{symbol}"
    data = await redis.get(key)
    if data:
        return json.loads(data)
    else:
        return None

Despite successful data queries (as per my logs), and ensuring that the queried data is not empty and that the used keys are consistent, the fetching function consistently returns None.

Why might this be occurring, and how can I effectively debug and resolve this issue?

Answer 1

I managed to solve the issue. The problem was related to network communication between Docker containers.

When running applications in Docker containers, each container has its own isolated network. So, when we try to connect to a Redis server using 'localhost', it tries to connect to a Redis server within the same container where the application is running.

To fix this, we need to correctly set up network communication between the containers. Here is what needs to be done:

1. Service Names in Docker Compose: In the docker-compose.yml file, the service names are used as hostnames for network communication between containers. In this case, the service name for Redis is 'redis'.

2. Using Service Names as Hostnames: In the code, instead of connecting to 'localhost', we should connect to 'redis' (the service name defined in the docker-compose.yml file).