How to fix memory leak of flask application with tracemalloc

Question

I have a flask application that makes requests to retrieve data and then exports the data as an excel file with openpyxl. After ~ 50 exported excel files the flask application exceeds the RAM of 8GB and crashes.

I have used tracemalloc with following code to find the memory leak. The function is called each time after an excel file is exported.

import tracemalloc
tracemalloc.start()

def get_allocated_memory():
   """
   Prints allocated memmory at time of function call in log file.
   """

   snapshot = tracemalloc.take_snapshot()
   top_stats = snapshot.statistics('lineno')

   logger.debug("========== ALLOCATED MEMMORY =============")
   for stat in top_stats[:10]:
      logger.debug(str(stat))
      logger.debug(str(stat.traceback.format()))

I get the following result after generating the first excel file (50MB), but could not fix any of the memory leaks successfully so far.

========== ALLOCATED MEMMORY =============
/Users/.../opt/anaconda3/lib/python3.9/json/decoder.py:353: size=108 MiB, count=994668, average=113 B
['  File "/Users/.../opt/anaconda3/lib/python3.9/json/decoder.py", line 353', '    obj, end = self.scan_once(s, idx)']
/Users/i.../PIR.py:147: size=7475 KiB, count=54234, average=141 B
['  File "/Users/.../PIR.py", line 147', '    dict_category[\'/glossary/\' + replaced] = entry[\'rep\'][\'title\'].replace("&", "&") + \'|\' + \\']
/Users/.../opt/anaconda3/lib/python3.9/ssl.py:1124: size=5120 KiB, count=1, average=5120 KiB
['  File "/Users/.../opt/anaconda3/lib/python3.9/ssl.py", line 1124', '    return self._sslobj.getpeercert(binary_form)']
/Users/.../PIR.py:143: size=4614 KiB, count=30535, average=155 B
['  File "/Users/...PIR.py", line 143', '    dict_category[\'/glossary/\' + entry[\'rep\'][\'id\']] = entry[\'rep\'][\'title\'].replace("&", "&") + \'|\' + \\']
/Users/.../PIR.py:155: size=3146 KiB, count=89489, average=36 B
['  File "/Users/.../PIR.py", line 155', "    replaced] = [entry['rep']['title'], entry['rep']]"]
/Users/.../PIR.py:151: size=1738 KiB, count=49430, average=36 B
['  File "/Users/i.../PIR.py", line 151', "    ] = [entry['rep']['title'], entry['rep']]"]
<frozen importlib....>:647: size=1473 KiB, count=17692, average=85 B
['  File "<frozen importlib._bootstrap_external>", line 647']
/Users/..../opt/anaconda3/lib/python3.9/site-packages/openpyxl/utils/cell.py:94: size=926 KiB, count=18252, average=52 B
['  File "/Users/.../opt/anaconda3/lib/python3.9/site-packages/openpyxl/utils/cell.py", line 94', "    return ''.join(reversed(letters))"]
/Users/.../opt/anaconda3/lib/python3.9/site-packages/openpyxl/descriptors/__init__.py:13: size=796 KiB, count=2685, average=304 B
['  File "/Users/.../opt/anaconda3/lib/python3.9/site-packages/openpyxl/descriptors/__init__.py", line 13', '    return type.__new__(cls, clsname, bases, methods)']
<frozen importlib._bootstrap>:228: size=673 KiB, count=6360, average=108 B
['  File "<frozen importlib._bootstrap>", line 228']

Any help and tips are highly appreciated!

Answer 1

Stab in the dark, are you using the built in Flask dev server? Try using something like Gunicorn (which you should be using in production anyway ;-)

gunicorn -w 4 -b 0.0.0.0 'hello:app'

There may of course just be some small error in the code, which could be tricky to track down. Also see this if you haven't yet : Flask App Memory Leak caused by each API call