Can you filter out personal details from Python Twilio logger?

Question

By default the Twilio Python client seems to log out personal information (emails and numbers). I know you can access the logger like this twilio_logger = logging.getLogger('twilio.http_client')

Is there a simple setting change we can make to filter out this personal information, either directly through the library or by changing the logger?

NOTE: I have attempted to do this by creating a custom formatter:

class SensitiveDataFormatter(logging.Formatter):
    """Formatter that removes sensitive information in urls."""
    @staticmethod
    def _filter(s):
        print('filter')
        return re.sub(r'([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,})', r'[email_redacted]', s)

    def format(self, record):
        original = logging.Formatter.format(self, record)
        return self._filter(original)


LOGGING = {
    'version': 1,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'sensitive'
        },
    },
    'root': {
        'handlers': ['console'],
        'level': 'INFO'
    },
    'formatters': {
        'sensitive': {
            '()': 'my_project.settings.logging.sensitive_data_formatter.SensitiveDataFormatter'
        }
    }
}

This works in the general case, however I'm struggling to apply it to Twilio

Answer 1

It seems that you can apply filters to Python loggers using logger config and a custom class that inherits from logging.Filter.

I'm not a Python developer, so pointing you to this blog post is the best way I can answer this question. I will note that the post appears to be from 2014, so may need updates for Python 3, but the Python logging documentation is also pretty extensive, so should help you out here.

Answer 2

Here is how you can configure the Twilio logger in the Python library to use a custom formatter. To test this I took your formatter and modified it to redact Twilio accounts, messaging services and phone numbers.

client = Client(account_sid, auth_token)

class SensitiveDataFormatter(logging.Formatter):
    @staticmethod
    def _filter(s):
        filters = [
            [r'(AC[a-f0-9]+)', '[account_redacted]'],
            [r'(MG[a-f0-9]+)', '[messaging_service_redacted]'],
            [r'(\+[0-9]+)', '[phone_redacted]'],
        ]
        for f in filters:
            s = re.sub(f[0], f[1], s)
        return s

    def format(self, record):
        original = logging.Formatter.format(self, record)
        return self._filter(original)

# create a logging handler with the custom formatter attached
handler = logging.StreamHandler()
handler.setFormatter(SensitiveDataFormatter())

# add the stream handler to the Twilio logger
client.http_client.logger.addHandler(handler)
client.http_client.logger.setLevel(logging.INFO)

Example log after sending an SMS:

POST Request: https://api.twilio.com/2010-04-01/Accounts/[account_redacted]/Messages.json
PAYLOAD: {'To': '[phone_redacted]', 'From': '[messaging_service_redacted]', 'Body': 'Friendly reminder that you have an appointment with us next week.'}
POST Response: 201 {"sid": "SM4428eb430949461d9bd58d44da1dd999", "date_created": "Thu, 19 May 2022 08:57:49 [phone_redacted]", "date_updated": "Thu, 19 May 2022 08:57:49 [phone_redacted]", "date_sent": null, "account_sid": "[account_redacted]", "to": "[phone_redacted]", "from": null, "messaging_service_sid": "[messaging_service_redacted]", "body": "Friendly reminder that you have an appointment with us next week.", "status": "accepted", "num_segments": "0", "num_media": "0", "direction": "outbound-api", "api_version": "2010-04-01", "price": null, "price_unit": null, "error_code": null, "error_message": null, "uri": "/2010-04-01/Accounts/[account_redacted]/Messages/SM4428eb430949461d9bd58d44da1dd999.json", "subresource_uris": {"media": "/2010-04-01/Accounts/[account_redacted]/Messages/SM4428eb430949461d9bd58d44da1dd999/Media.json"}}