RoundRobin by Time Window

Question

I'm looking for a system that stores timestamped values and remove automatically the ones that are older than specific time window.

Something like a RoundRobin system but, as long I understand, RoundRobin is based in an specific amount of records and removing as much old ones as much new ones are coming.

In this system can be any amount of records and just removing the ones that are older than an specific time lapsus.

One approximation can be a system where every record has a Timeout live like:

console.append "key", { :value => "value1", :timeout => 10.minutes_in_the_future }
console.append "key", { :value => "value2", :timeout => 10.minutes_in_the_future }
console.append "key", { :value => "value3", :timeout => 10.minutes_in_the_future }

And somehow the system is in charge to remove any expired record.

The system can be based in any high-performance storage system (Redis, Memcache, ...)

I'm not looking for a full detailed explanation implementation just some kind of inspiration or any related article or already existing system so I'm not reinventing the wheel.

Answer 1

Memcache aprox

In the office we are studying this aprox:

Preconditions

• The time window is 10 minutes.
• We only need to know the amount of records and not the values.
• We can live with not very accurate solution.

Implementation

1. Create a new key every minute like: mykey_<Time.now.strftime("%Y%m%d%H%M")>.
2. Give them a expire time of 10 minutes.
3. Every time we want to add a record we incr the key for our actual minute.
4. To calculate the time window value we get the values of the 10 keys for the last 10 minutes and sum them.

Cons

• We can not store real values but only one accumulate.
• When every 1 minute key is expired the total value fluctuates suddenly.