Vanity A/B testing with page caching

Question

I'm using Vanity to do my A/B testing but I also have page caching for the pages the experiment is running on. Because of that, the user will actually see both options as they navigate around the site. I would be fine with that however I know that Vanity keeps track of which option a single user is seeing and attempts to show that same option throughout the site.

So what I'm wondering is, let's say a user comes to my site and views Option A. Vanity (I think) records them as an "Option A" participant. Now that user goes to a different cached page, and because of the caching they see Option B and "converts".

Is vanity going to record that user as an "Option A" conversion or an "Option B" conversion?

Answer 1

I've recently started helping to maintain Vanity.

This question is similar to the issue reported here, which was recently addressed and closed by PR#144 (just merged onto master, will be included in the next release).

A cache string for a user can be generated using (from a controller):

Vanity.playground.participant_info(vanity_identity)

Answer 2

I spoke with the creator of Vanity and it is indeed important to turn off caching of pages you're trying to A/B test with in order to get accurate results.