uWSGI / Flask / Python logs stop after some time

Question

I have a uWSGI / Flask setup using python loggers. Though logs only from some workers get to the logs and after some time even those cease to show up at all. My hypothesis is that when uWSGI restarts (clones) workers, logging somehow gets broken. Any ideas?

app/server.py:

app = Flask(...)
handler = logging.StreamHandler()
app.logger.addHandler(handler)
app.run()

uWSGI:

uwsgi --emperor /etc/uwsgi/apps-enabled/*.ini --die-on-term --uid www-data --gid www-data --logto /var/www/app.com/logs/uwsgi/emperor.log --socket /tmp/uwsgi/emperor.sock --enable-threads --master --single-interpreter --log-reopen --chmod-socket=770

apps-enabled/app-0.ini and apps-enabled/app-1.ini look like this:

module=server:app
enable-threads=true
single-interpreter=true
master=true
chdir=/var/www/app.com/app
env=APPLICATION_ENVIRONMENT=production
venv=/var/www/app.com/virtualenv

logto=/var/www/app.com/logs/uwsgi/app.com-0.log
log-reopen=true
chmod-socket=770
buffer-size=65535

lazy-apps=true
max-requests=5000
heartbeat=15

for=0 1 2 3 4 5 6 7
socket=/tmp/uwsgi/app.0.%(_).sock
endfor=

processes=8

map-socket=0:1
map-socket=1:2
map-socket=2:3
map-socket=3:4
map-socket=4:5
map=socket=5:6
map=socket=6:7
map=socket=7:8

I have also tried to use SysLogHandler with the same result.

Answer 1

I can tell you for sure is that restarting uwsgi should not stop logging. I have nginx-uwsgi-django server and I restart my uwsgi server all the time and my logs never stop. Here is what my ini file looks like: may be you can tweak your ini file as per this and see if it works.

1 #mysite_uwsgi.ini
2 [uwsgi]
3 
4 # Django-related settings
5 # the base directory (full path)
6 chdir           = /home/user/bdapps_stage
7 # Django's wsgi file
8 module          = mysite.wsgi:application
9 # the virtualenv (full path)
10 home            = /home/user/.conda/envs/mysite_env/
11 
12 # process-related settings
13 # master
14 master          = true
15 # maximum number of worker processes
16 processes       = 3
17 # maximum number of threads to use
18 # threads
19 # the socket (use the full path to be safe
20 socket          = /home/user/mysite/mysite.sock
21 # ... with appropriate permissions - may be needed
22 chmod-socket    = 666
23 #set the sockets listen queue size
24 #listen
25 # clear environment on exit
26 vacuum          = true

And here is how I restart my uwsgi

kill -SIGHUP [pid id of your uwsgi master]

Please note that the command you have provided for uwsgi using the ini file should only be used once to start the uwsgi server. When you want to restart, I suggest you note down the pid of the uwsgi master by using

 ps -ef | grep uwsgi 

and run the above kill command.

Answer 2

I'm not sure that this will help in your case, but you can try to use post forking in uWSGI.

The postfork decorator is just the ticket. You can declare multiple postfork tasks. Each decorated function will be executed in sequence after each fork().

@postfork
def init_logging():
    app.logger.addHandler(handler)

Or you can specify lazy-apps=true in your uwsgi.ini.

For details: http://uwsgi-docs.readthedocs.io/en/latest/PythonDecorators.html and http://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html

Answer 3

Venturing another possible cause for logging stopping after some time. uWSGI can be set to drop privileges (which is a good idea). However after this event logto2 [0] setting is used:

logto=/var/www/app.com/logs/uwsgi/app.com-0.log
logto2=/var/www/app.com/logs/uwsgi/app.com-normaluser-0.log

Hope it helps someone.

[0] https://uwsgi-docs.readthedocs.io/en/latest/Options.html#logto2