Django-compressor has a JS cache folder that is using an absurd amount of space

Question

I woke up this morning to alerts from sentry that my production server has completely run out of space. It took some time figuring out the cause via ncdu, and the results were that my static folder had used over 60GB of space, specifically, CACHE/js that django-compressor is using.

I am not completely sure what is happening, or why there are over 500,000 js files where each file is following this format: output.<random string>.js. From my understanding, shouldn't there only be a small number of js files cached? My project doesn't even have that many scripts!

It seems to me that every user is getting their own output file, instead of the same cached files being shared to everyone.

Base settings:

# STATIC
# ------------------------------------------------------------------------------
# https://docs.djangoproject.com/en/dev/ref/settings/#static-root
# STATIC_ROOT = str(ROOT_DIR / "static")
STATIC_ROOT = os.path.join(BASE_DIR, "static/")
# https://docs.djangoproject.com/en/dev/ref/settings/#static-url
STATIC_URL = "/static/"
# https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#std:setting-STATICFILES_DIRS
# STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")]
# https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#staticfiles-finders
STATICFILES_FINDERS = [
    "django.contrib.staticfiles.finders.FileSystemFinder",
    "django.contrib.staticfiles.finders.AppDirectoriesFinder",
    "compressor.finders.CompressorFinder",
]

COMPRESS_ENABLED = True
COMPRESS_PRECOMPILERS = (
    ('text/x-scss', 'django_libsass.SassCompiler'),
)
COMPRESS_FILTERS = {
    "css": [
        'compressor.filters.css_default.CssAbsoluteFilter',
        # 'compressor.filters.cssmin.CSSMinFilter',
        'core.CSSMinFilter.CSSMinFilter',
    ]
}

production settings:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

I originally installed django-compressor (https://django-compressor.readthedocs.io/en/stable/quickstart/) to fix issues where users would have to manually clear their browser cache to see correct changes to templates.

Any ideas on what is going on, or how to fix this?

edit:

I ran the following command rm -rf js to delete all the files, and within like a minute there's already 200 files created.

Answer 1

Search in your codebase for "compress js". It should be present in your django templates (.html files). There should also be an "endcompress" tag below it. If there are any context variables passed in between these tags, try to put them outside. For e.g. in the below code, {{ username }} and {{ userid }} and are inside the compress tags.

Before:

{% compress js %}
<script src="{% static 'include/jquery.js' %}"></script>
<script>
    var username = {{ username }};
    var userid = {{ userid }};
    ...
</script>
{% endcompress %}

After:

{% compress js %}
<script src="{% static 'include/jquery.js' %}"></script>
{% endcompress %}

<script>
    var username = {{ username }};
    var userid = {{ userid }};
    ...
</script>

As @user12758446 commented, these context variables are changing on every request. Each time, the django-compressor module will create a new file for the changed value. You can check its documentation here. So just ensure that whatever you are putting inside the {% compress js %} and {% endcompress %} is static.