Learning to rank using Lenskit

Question

I am using lenskit framework for learning to rank problem. I have an issue that I have never seen anybody reporting.
When I compare performance of FunkSVD and Popularity baseline on the whole universe of items, popularity baseline significantly outperforms FunkSVD. I think it is cased by the fact the popular items are rated more often, while it does not mean that users like popular items.
I limited number of candidate items for evaluation to N popular ones. As a result, the performance of FunkSVD increased, since the algorithm has less opportunity to make a mistake. FunkSVD outperformed popularity. However, now I have another issue. Random baseline beats both FunkSVD and popularity! I guess that's because the chance to make a mistake is very low, but I am conduced. Is there a way to make FunkSVD perform better than popularity and random? Am I doing anything wrong? Is the problem in the framework? Has anybody encountered this problem?

Answer 1

How are you measuring performance? That is a key thing that will make a significant difference.

Also, FunkSVD has a number of parameters that must be tuned in order to have good performance. Default values may well not be good for your data set.

Diagnosing exactly what is going wrong with a specific experiment seems like something better handled in the LensKit discussion fora (mailing list or Gitter room).