How to low stdout verbosity for AppDynamics Agent and Proxy in Python Applications

Question

I'm integrating the AppDynamics Python Agent into a FastAPI project for monitoring purposes and have encountered a bit of a snag regarding log verbosity in my stdout. I'm launching my FastAPI app with the following command to include the AppDynamics agent:

pyagent run -c appdynamics.cfg uvicorn my_app:app --reload

My goal is to reduce the verbosity of the logs from both the AppDynamics agent and the proxy that are output to stdout, aiming to keep my console output clean and focused on more critical issues.

My module versions:

$ pip freeze | grep appdy
appdynamics==23.10.0.6327
appdynamics-bindeps-linux-x64==23.10.0
appdynamics-proxysupport-linux-x64==11.64.3

Here's the content of my appdynamics.cfg configuration file:

[agent]
app = my-app
tier = my-tier
node = teste-local-01

[controller]
host = my-controller.saas.appdynamics.com
port = 443
ssl = true
account = my-account
accesskey = my-key

[log]
level = warning
debugging = off

I attempted to decrease the log verbosity further by modifying the log4j.xml file for the proxy to set the logging level to WARNING. However, this change didn't have the effect I was hoping for. The log4j.xml file I adjusted is located at:

/tmp/appd/lib/cp311-cp311-63ff661bc175896c1717899ca23edc8f5fa87629d9e3bcd02cf4303ea4836f9f/site-packages/appdynamics_bindeps/proxy/conf/logging/log4j.xml

Here are the adjustments I made to the log4j.xml:

    <appender class="com.singularity.util.org.apache.log4j.ConsoleAppender" name="ConsoleAppender">
        <layout class="com.singularity.util.org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d{ABSOLUTE} %5p [%t] %c{1} - %m%n" />
        </layout>
        <filter class="com.singularity.util.org.apache.log4j.varia.LevelRangeFilter">
            <param name="LevelMax" value="FATAL" />
            <param name="LevelMin" value="WARNING" />
        </filter>

Despite these efforts, I'm still seeing a high volume of logs from both the agent and proxy. Could anyone provide guidance or suggestions on how to effectively lower the log output to stdout for both the AppDynamics Python Agent and its proxy? Any tips on ensuring my changes to log4j.xml are correctly applied would also be greatly appreciated.

Thank you in advance for your help!

Example of logging messages I would like to remove from my stdout:

2024-03-23 11:15:28,409 [INFO] appdynamics.proxy.watchdog <22759>: Started watchdog with pid=22759
2024-03-23 11:15:28,409 [INFO] appdynamics.proxy.watchdog <22759>: Started watchdog with pid=22759
...
[AD Thread Pool-ProxyControlReq0] Sat Mar 23 11:15:51 BRT 2024[DEBUG]: JavaAgent - Setting AgentClassLoader as Context ClassLoader
[AD Thread Pool-ProxyControlReq0] Sat Mar 23 11:15:52 BRT 2024[INFO]: JavaAgent - Low Entropy Mode: Attempting to swap to non-blocking PRNG algorithm
[AD Thread Pool-ProxyControlReq0] Sat Mar 23 11:15:52 BRT 2024[INFO]: JavaAgent - UUIDPool size is 10
Agent conf directory set to [/home/wsl/.pyenv/versions/3.11.6/lib/python3.11/site-packages/appdynamics_bindeps/proxy/conf]
...
11:15:52,167  INFO [AD Thread Pool-ProxyControlReq0] BusinessTransactions - Starting BT Logs at Sat Mar 23 11:15:52 BRT 2024
11:15:52,168  INFO [AD Thread Pool-ProxyControlReq0] BusinessTransactions - ###########################################################
11:15:52,169  INFO [AD Thread Pool-ProxyControlReq0] BusinessTransactions - Using Proxy Version [Python Agent v23.10.0.6327 (proxy v23.10.0.35234) compatible with 4.5.0.21130 Python Version 3.11.6]
11:15:52,169  INFO [AD Thread Pool-ProxyControlReq0] JavaAgent - Logging set up for log4j2
...
11:15:52,965  INFO [AD Thread Pool-ProxyControlReq0] JDBCConfiguration - Setting normalizePreparedStatements to true
11:15:52,965  INFO [AD Thread Pool-ProxyControlReq0] CallGraphConfigHandler - Call Graph Config Changed  callgraph-granularity-in-ms  Value -null

Answer 1

I came across a workaround that was suggested in an unofficial capacity by someone at AppDynamics during their local lab explorations. While this solution isn't officially supported by AppDynamics, it has proven to be effective for adjusting the log levels for both the Proxy and the Watchdog components within my AppDynamics setup. I'd like to share the steps involved, but please proceed with caution and understand that this is not a sanctioned solution.

I recommend changing only the log4j2.xml file, because the proxy messages look like are responsible for almost 99% of the log messages.

Here's a summary of the steps:

• Proxy Log Level: The log4j2.xml file controls this. You can find it within the appdynamics_bindeps module. For example, in my WSL setup, it's located at /home/wsl/.pyenv/versions/3.11.6/lib/python3.11/site-packages/appdynamics_bindeps/proxy/conf/logging/log4j2.xml. In the Docker image python:3.9, the path is /usr/local/lib/python3.9/site-packages/appdynamics_bindeps/proxy/conf/logging/log4j2.xml. Modify the seven log level itens <AsyncLogger> within the <Loggers> section to one of the following: debug, info, warn, error, or fatal.

• Watch Dog Log Level: This can be adjusted in the proxy.py file found within the appdynamics Python module. For example, in my WSL setup, it's located at /home/wsl/.pyenv/versions/3.11.6/lib/python3.11/site-packages/appdynamics/scripts/pyagent/commands/proxy.py. In the Docker image python:3.9, the path is /usr/local/lib/python3.9/site-packages/appdynamics/scripts/pyagent/commands/proxy.py. You will need to hardcode the log level in the configure_proxy_logger and configure_watchdog_logger functions by changing the level variable.

My versions

$ pip freeze | grep appdynamics
appdynamics==24.2.0.6567
appdynamics-bindeps-linux-x64==24.2.0
appdynamics-proxysupport-linux-x64==11.68.3

Original files

log4j2.xml

<Loggers>
    <!-- Modify each <AsyncLogger> level as needed -->
        <AsyncLogger name="com.singularity" level="info" additivity="false">
            <AppenderRef ref="Default"/>
            <AppenderRef ref="RESTAppender"/>
            <AppenderRef ref="Console"/>
        </AsyncLogger>
</Loggers>

proxy.py

def configure_proxy_logger(debug):
    logger = logging.getLogger('appdynamics.proxy')
    level = logging.DEBUG if debug else logging.INFO
    pass

def configure_watchdog_logger(debug):
    logger = logging.getLogger('appdynamics.proxy')
    level = logging.DEBUG if debug else logging.INFO
    pass

My Script to create environment variables to log4j2.xml and proxy.py

update_appdynamics_log_level.sh

#!/bin/sh

# Check if PYENV_ROOT is not set
if [ -z "$PYENV_ROOT" ]; then
    # If PYENV_ROOT is not set, then set it to the default value
    export PYENV_ROOT="/usr/local/lib"
    echo "PYENV_ROOT was not set. Setting it to default: $PYENV_ROOT"
else
    echo "PYENV_ROOT is already set to: $PYENV_ROOT"
fi

echo "=========================== log4j2 - appdynamics_bindeps module ========================="

# Find the appdynamics_bindeps directory
APP_APPD_BINDEPS_DIR=$(find "$PYENV_ROOT" -type d -name "appdynamics_bindeps" -print -quit)

if [ -z "$APP_APPD_BINDEPS_DIR" ]; then
  echo "Error: appdynamics_bindeps directory not found."
  exit 1
fi

echo "Found appdynamics_bindeps directory at $APP_APPD_BINDEPS_DIR"

# Find the log4j2.xml file within the appdynamics_bindeps directory
APP_LOG4J2_FILE=$(find "$APP_APPD_BINDEPS_DIR" -type f -name "log4j2.xml" -print -quit)

if [ -z "$APP_LOG4J2_FILE" ]; then
  echo "Error: log4j2.xml file not found within the appdynamics_bindeps directory."
  exit 1
fi

echo "Found log4j2.xml file at $APP_LOG4J2_FILE"

# Modify the log level in the log4j2.xml file
echo "Modifying log level in log4j2.xml file"
sed -i 's/level="info"/level="${env:APP_APPD_LOG4J2_LOG_LEVEL:-info}"/g' "$APP_LOG4J2_FILE"

echo "log4j2.xml file modified successfully."

echo "=========================== watchdog - appdynamics module ==============================="

# Find the appdynamics directory
APP_APPD_DIR=$(find "$PYENV_ROOT" -type d -name "appdynamics" -print -quit)

if [ -z "$APP_APPD_DIR" ]; then
  echo "Error: appdynamics directory not found."
  exit 1
fi

echo "Found appdynamics directory at $APP_APPD_DIR"

# Find the proxy.py file within the appdynamics directory
APP_PROXY_PY_FILE=$(find "$APP_APPD_DIR" -type f -name "proxy.py" -print -quit)

if [ -z "$APP_PROXY_PY_FILE" ]; then
  echo "Error: proxy.py file not found within the appdynamics directory."
  exit 1
fi

echo "Found proxy.py file at $APP_PROXY_PY_FILE"

# Modify the log level in the proxy.py file
echo "Modifying log level in proxy.py file"
sed -i 's/logging.DEBUG if debug else logging.INFO/os.getenv("APP_APPD_WATCHDOG_LOG_LEVEL", "info").upper()/g' "$APP_PROXY_PY_FILE"


echo "proxy.py file modified successfully."

Dockerfile

Dockerfile to run pyagent with FastAPI and run this script

# Use a specific version of the python image
FROM python:3.9

# Set the working directory in the container
WORKDIR /app

# First, copy only the requirements file and install dependencies to leverage Docker cache
COPY requirements.txt ./
RUN python3 -m pip install --no-cache-dir -r requirements.txt

# Now copy the rest of the application to the container
COPY . .

# Make the update_appdynamics_log_level.sh executable and run it
RUN chmod +x update_appdynamics_log_level.sh && \
    ./update_appdynamics_log_level.sh 

# Set environment variables
ENV APP_APPD_LOG4J2_LOG_LEVEL="warn" \
    APP_APPD_WATCHDOG_LOG_LEVEL="warn"

EXPOSE 8000

# Command to run the FastAPI application with pyagent
CMD ["pyagent", "run", "uvicorn", "main:app", "--proxy-headers", "--host","0.0.0.0", "--port","8000"]

Files changed by the script

log4j2.xml

<Loggers>
    <!-- Modify each <AsyncLogger> level as needed -->
        <AsyncLogger name="com.singularity" level="${env:APP_APPD_LOG4J2_LOG_LEVEL:-info}" additivity="false">
            <AppenderRef ref="Default"/>
            <AppenderRef ref="RESTAppender"/>
            <AppenderRef ref="Console"/>
        </AsyncLogger>
</Loggers>

proxy.py

def configure_proxy_logger(debug):
    logger = logging.getLogger('appdynamics.proxy')
    level = os.getenv("APP_APPD_WATCHDOG_LOG_LEVEL", "info").upper()
    pass

def configure_watchdog_logger(debug):
    logger = logging.getLogger('appdynamics.proxy')
    level = os.getenv("APP_APPD_WATCHDOG_LOG_LEVEL", "info").upper()
    pass

Warning

Please note, these paths and methods may vary based on your AppDynamics version and environment setup. Always backup files before making changes and be aware that updates to AppDynamics may overwrite your customizations.

I hope this helps!