Varnish : Make An API request to fetch data and embedd it into response header to client

Question

I want to send a cached page back to the user But the problem is that I need to generate a unique VISITOR_ID for every new user and send it back to the user through headers , so I need to send an API call from varnish proxy server to my backend servers to fetch VISITOR_ID and then append it to the response

We were earlier using Akamai and we were able to implement this using edge workers present there, I want to know if such a thing is possible to do in varnish or not.

Thanks in Advance

Answer 1

Open source solution for HTTP calls

You can use https://github.com/varnish/libvmod-curl and add this VMOD to perform HTTP calls from within VCL. The API for this module can be found here: https://github.com/varnish/libvmod-curl/blob/master/src/vmod_curl.vcc

Commercial solution for HTTP calls

Despite there being an open source solution, I want to mention that there might be a more stable solution that is actually supported and receives frequent updates. See https://docs.varnish-software.com/varnish-cache-plus/vmods/http/ for the HTTP VMOD that is part of Varnish Enterprise.

Generate the VISITOR_ID in VCL

Your solution implies that an HTTP call is needed for every single response. While that is possible through various VMODs, this will result in a lot of extra HTTP calls. Unless you cache every variation that includes the VISITOR_ID.

You could also consider generating the unique ID yourself in VCL.

See https://github.com/otto-de/libvmod-uuid for a VMOD that generates UUIDs or https://github.com/varnish/libvmod-digest for a VMOD that generates hashes.

Fetch VISITOR_ID from Redis

If you prefer to generate the VISITOR_ID in your origin application, you could use a Key/Value store like Redis to store or generate values.

You can generate the ID in your application and store it in Redis. You could also generate and store it using LUA scripting in Redis.

Varnish can then fetch the key from Redis and inject it in the response.

While this is a similar approach to the HTTP calls, at leasts we know Redis is capable of keeping up with Varnish in terms of high performance.

See https://github.com/carlosabalde/libvmod-redis to learn how to interface with Redis from Varnish.

Answer 2

Are you sure you need to make an API call to the backend?

One common approach to this problem is to use a Cookie to store the VISITOR_ID. In Varnish, you could have VCL that looks like this:

sub vcl_recv {
    if (req.http.Cookie !~ 'VISITOR_ID') {
        # go straight to the backend, bypassing cache.
        return (pass);
    }
    unset req.http.Cookie;
}

sub vcl_backend_fetch {
    # for a return (pass), beresp.uncacheable will be set to 'true' at the top of vcl_backend_fetch
    if (beresp.uncacheable != 'true') {
        unset beresp.http.Set-Cookie;
    }
}

The idea here is that on the first request, when the VISITOR_ID cookie is not yet set, the client request will go to the backend, and a Set-Cookie will be in the response. On subsequent requests, the Client will be sending the VISITOR_ID cookie, which will allow the request get it's Cookie header unset in vcl_recv and be a cache hit.

We are also unsetting the Set-Cookie response header in vcl_backend_recv, because on cache misses, the request from Varnish to the Origin will not contain a Cookie header, so we expect the backend to include a Set-Cookie for VISITOR_ID in the response. This would cause Varnish by default not to cache this page, unless we unset the Set-Cookie header.

Instead of allowing the backend to set the Visitor ID, would it be possible to allow Varnish to do this? The benefit to this approach is that the Client can get a cache hit from the first request, which will be a lot lower latency than a backend fetch. This could be done in VCL by setting a unique Cookie in vcl_deliver, using a unique hash instead of using your backend to do this.

sub vcl_deliver {
    set resp.http.Set-Cookie = 'UNIQUE_ID=' + <unique_id>;
}

Where <unique_id> could be generated using vmod blobsha256 and the request id.

Considering your requirements, if you are serving content from cache while having a unique visitor Id from the backend, you may be thinking about an attribution model. For uncacheable content (such as a purchase), the backend will receive cookies from the client, which will contain the unique VISITOR ID. However, the backend will not know about any cache hits the client received from Varnish. If this is relevant to collect, you may want to use varnishncsa with a custom format string, and an infrastructure agent from a data aggregation tool like Grafana, so you can track visitor's path through your website, even including cache hits.