Is it possible to create two layers of multi-tiered cache based on memory -> disk cache layers on the same machine?

Question

Is it possible to create two layers of multi-tiered cache based on memory -> disk cache layers on the same machine? We would like to create two layers of tiered cache based on memory -> disk with the open sourced version of varnish 6+. The idea is to create two storage back-ends (which is possible to do) with memory and disk on the same server. The disk tier is about 1TB of very fast SSD and the memory tier is about 256G.

My idea is to create some sort of hot (memory - for extremely popular content) and cold (disk - for less popular content) cache on the same server, which have different TTLs. For example, if an item has too many hits on specific file, it will move to memory for at least 5 to 10 seconds and if the item doesn't exist on the memory tier, it will go to the local disk and fetch it from the SSD tier (cold).

The problem is that I can't find the best way to implement the logic inside vcl_backend_responce. It is possible to set a condition based on host or files types, but I can't find anywhere how to redirect the request from memory storage to disk storage before it goes out the remote HTTP server to fetch the file.

Below is the idea of how I think it should be implemented, the only missing part is what condition should be inside the if statements. I would appreciate your inputs or ideas if there are any.

The place the code should be placed -> vcl_backend_responce

if (obj.hits exsit on beresp.storage_hint ="memory") {
    set beresp.storage_hint ="memory"
    set beresp.ttl = 5s;
}elseif (obj.hits exsit on beresp.storage_hint ="disk") {
    set beresp.storage_hint ="disk"
    set beresp.ttl = 2m;
}

enter image description here

File -->Disk --Memory --> client
         2M (DISK TTL )      10S (Memory TTL) 

Answer 1

You may use multiple varnish ports to achieve that.

    ExecStart=/usr/sbin/varnishd \
      -a :6081 \
      -a :6181 \
      -T localhost:6082 \
      -f /etc/varnish/default.vcl \
      -P %t/%N/varnishd.pid \
      -p feature=+http2 \
      -s memory=malloc,5G \
      -s file=file,/mnt/cache/data,100G

Above command allows Varnish to run on both 6081 and 6181. Then yo can manage storage usage in your vcl with simple if/else condition.

sub vcl_recv
{
  if (std.port(server.ip) == 6081) {
    set req.backend_hint = cluster.backend();

    # If backend hits, pass to layer2Cache which is 6181
    if (req.backend_hint == server.identity) { 
      set req.backend_hint = layer2Cache;
    } else {
      return(pass);
   }
  } else {
    // if it is layer2Cache, pass to origin
    set req.backend_hint = origin; 
  }
}

sub vcl_backend_response {
  if (std.port(server.ip) == 6081) {
    set beresp.storage = storage.memory;
      
    set beresp.ttl = 60s;
    set beresp.grace = 1s * 60 * 5;
  } else {
    set beresp.storage = storage.file;
    
    set beresp.ttl = 240s;
    set beresp.grace = 1s * 60 * 60 * 24;
  }
}

We are using this method in our cache servers, but please note that the code above may not work with direct copy/paste.

Answer 2

Please don't use the file storage engine, because it will slow down your system and is very sensitive to disk fragmentation.

There are 3 alternative solutions I'd like to present.

Memory-only tiered storage on multiple nodes

If is it possible for you to host Varnish on multiple servers, you could create a memory-based tiered architecture that scales well.

By using the sharding directory every cache miss from the first tier will always end up on the same node on the second tier. This allows you to horizontally scale your second tier by just adding servers.

The first tier would handle all the hot objects and long tail content would end up on the second tier. This means your second tier acts as the storage tier.

Here's a VCL snippet for the routing on your first tier:

vcl 4.1;

import directors;

backend storage1 {
    .host = "storage1.example.com";
    .port = "80";
}

backend storage2 {
    .host = "storage2.example.com";
    .port = "80";
}

backend storage3 {
    .host = "storage3.example.com";
    .port = "80";
}

sub vcl_init {
    new vdir = directors.shard();
    vdir.add_backend(storage1);
    vdir.add_backend(storage2);
    vdir.add_backend(storage3);
    vdir.reconfigure();
}

sub vcl_backend_fetch {
    set bereq.backend = vdir.backend();
}

This is a very flexible and scalable setup. The downside is that it requires more hardware.

Massive Storage Engine on Varnish Enterprise

Because of a poor file storage engine implementation in Varnish Cache, we at Varnish Software built a better alternative.

It's called Massive Storage Engine and is part of our Varnish Enterprise offering. See https://docs.varnish-software.com/varnish-enterprise/features/mse/ for more information.

IMPORTANT: this MSE feature is not compatible with open source Varnish Cache and requires a Varnish Enterprise license.

In MSE you can define various storage backends and a memory storage. Here's an example config:

env: {
    id = "mse";
    memcache_size = "auto";

    books = ( {
        id = "book";
        directory = "/var/lib/mse/book";
        database_size = "2G";

        stores = ( {
            id = "store1";
            filename = "/var/lib/mse/store1.dat";
            size = "100G";
            tags = ( "fast" );
        },{
            id = "store2";
            filename = "/var/lib/mse/store2.dat";
            size = "900G";
            tags = ( "slow" );
        } );
    } );
};

This is hosted in a separate config file that is mounted through the -s parameter in varnishd.

These storage files are pre-allocated large files that can take up most of your disk space. Inside these files MSE will have filesystem-like behavior that improves performance and actively avoids disk fragmentation.

The memory size is variable and MSE's built-in Memory Governor will ensure that the memory footprint of Varnish stays stable. If a lot of request are coming in, more memory is allocated to the threads and the memory cache will slightly shrink until there's enough memory available again. This is an important stability feature.

MSE also comes with a VMOD that allows you to perform ad hoc storage selections. Here's an example:

vcl 4.1;

import mse;
import std;

sub vcl_backend_response {
    if (beresp.ttl < 120s) {
        mse.set_stores("none");
    } else {
        if (beresp.http.Transfer-Encoding ~ "chunked" || 
        std.integer(beresp.http.Content-Length,0) > std.bytes("100M")) {
            mse.set_stores("fast");
        } else {
            mse.set_stores("slow");
        }
    }
}

The none store means that the data will not be persisted and will only be kept in memory. The tags in the MSE configuration file can be called through mse.set_stores() to select which disk the content will be stored on.

You have the flexibility to choose either RAM or disk. This implementation is tremendously fast and used by a lot of big companies. Of course, as mentioned, this is not a free and open source implementation.

Fellow & buddy storage engines by Uplex

The people at Uplex recently released their new storage engine, which is also a reaction on the poor performance of the file storage engine.

It is free and open source, however I haven't worked with it yet.

See https://gitlab.com/uplex/varnish/slash/-/tree/master/ for more information.

See https://gitlab.com/uplex/varnish/slash/-/blob/master/src/vmod_slash.man.rst for documentation on how to set it up and how to choose storage locations.

Answer 3

I just wanted to update this ticket regarding a test we have made in production with new slash engine . It seems that the storage engine still not matured enough ( at least for us ) . The storage engine ( slash ) causes many panic crushes all details here :

https://gitlab.com/uplex/varnish/slash/-/issues/7#note_1329659634

Regards