Best practice while saving data in HttpApplication.Application

Question

So I have a web service that returns data from a certain data source. Some of the data retrievals takes really long (>15 sec) which is unacceptable while serving data to a web page via ajax calls.

I was thinking that I can cache the data in the following method:

Run a worker thread in Global.asax that retrieves deltas every X minutes/hours and updates a variable, which will then be returned in the web service. The best solution that came to my mind was saving the data in the Application dictionary.

Basically my question is, should I do it? (We are talking about data that can exceed 100MB) Are there any ramifications doing it? If so, what solution will be best?

Answer 1

You're probably better off using caching for this case, as it will automatically be flushed out if the server is running low on memory.

See 'Application Considerations' here: https://msdn.microsoft.com/en-us/library/ms178594.aspx

Answer 2

It really depends on the nature of your data. Caching is the perfect solution for the data that doesn't change very often and is somewhat static. You can also use a short time caching for the data that is dynamic (changes often), but it is ok if user is using the data that is a little bit old. .Net framework has a CacheManager class, which lets you to define expiry for the cached data. I'd say instead of reading ALL data and saving them in the memory, using .Net cache manager, just cache the data that is requested for x minutes.

You can write your own code and get very creative on when and how to refresh your cached data. For example, you could add a counter property to the cached data and count how many time the data was accessed from cache, then either refresh the data of expand the lifetime of the cached data by extending its expiry timestamp.

Take a look at this link for start: http://cachemanager.net/Documentation/Index/cachemanager_getting_started