I am dealing with a recommendation problem, which involves 3 million users and 500,000 products. The purpose of the recommendation is to recommend 5-10 more products to the particular user when visiting a given product.
We finished the real-time recommendation engine part, but we found that it is not so easy to recommend in 1-2 seconds. We consider two strategies:
Run the recommendation off-line in a batch mode, then store the results in a MySQL DB. However, it seems like that the number of entries in the tables is a huge number. Each user for each product should store at least 5-10 recommended products. If I create a table with user and product as table attributes, there will be 3 million x 500,000 entries.
Predict the recommendation on the fly. We just tried it, but the response time is too long.
Can anyone give me some suggestions on this problem?
It looks strange to me to create a list of recommended products for each user-product-pair. Normally you only have a list of similar products for each product for use cases like yours.