stop memcache from removing data in php/silverstripe/zend cache

Question

I have recently installed memcache(d), but the data only seems to stay in the cache for a minute or so, even though I have set the expiration to three days. I am trying to cache 1000s of items, but only a few seem to stick! Here are the stats:

STAT pid 31586
STAT uptime 253023
STAT time 1480889337
STAT version 1.4.14 (Ubuntu)
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 17.270251
STAT rusage_system 72.982928
STAT curr_connections 5
STAT total_connections 9083
STAT connection_structures 15
STAT reserved_fds 20
STAT cmd_get 577101
STAT cmd_set 217153
STAT cmd_flush 19612
STAT cmd_touch 0
STAT get_hits 382249
STAT get_misses 194852
STAT delete_misses 10
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 544127879
STAT bytes_written 1028900006
STAT limit_maxbytes 536870912
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 524288
STAT hash_is_expanding 0
STAT expired_unfetched 0
STAT evicted_unfetched 0
STAT bytes 434350
STAT curr_items 163
STAT total_items 217153
STAT evictions 0
STAT reclaimed 111319

and here are the stats for the items:

stats items
STAT items:3:number 1
STAT items:3:age 133084
STAT items:3:evicted 0
STAT items:3:evicted_nonzero 0
STAT items:3:evicted_time 0
STAT items:3:outofmemory 0
STAT items:3:tailrepairs 0
STAT items:3:reclaimed 0
STAT items:3:expired_unfetched 0
STAT items:3:evicted_unfetched 0
STAT items:4:number 4
STAT items:4:age 857
STAT items:4:evicted 0
STAT items:4:evicted_nonzero 0
STAT items:4:evicted_time 0
STAT items:4:outofmemory 0
STAT items:4:tailrepairs 0
STAT items:4:reclaimed 616
STAT items:4:expired_unfetched 0
STAT items:4:evicted_unfetched 0
STAT items:5:number 2
STAT items:5:age 52
STAT items:5:evicted 0
STAT items:5:evicted_nonzero 0
STAT items:5:evicted_time 0
STAT items:5:outofmemory 0
STAT items:5:tailrepairs 0
STAT items:5:reclaimed 1364
STAT items:5:expired_unfetched 0
STAT items:5:evicted_unfetched 0
STAT items:6:number 39
STAT items:6:age 46
STAT items:6:evicted 0
STAT items:6:evicted_nonzero 0
STAT items:6:evicted_time 0
STAT items:6:outofmemory 0
STAT items:6:tailrepairs 0
STAT items:6:reclaimed 17782
STAT items:6:expired_unfetched 0
STAT items:6:evicted_unfetched 0
STAT items:7:number 55
STAT items:7:age 51
STAT items:7:evicted 0
STAT items:7:evicted_nonzero 0
STAT items:7:evicted_time 0
STAT items:7:outofmemory 0
STAT items:7:tailrepairs 0
STAT items:7:reclaimed 15681
STAT items:7:expired_unfetched 0
STAT items:7:evicted_unfetched 0
STAT items:8:number 24
STAT items:8:age 51
STAT items:8:evicted 0
STAT items:8:evicted_nonzero 0
STAT items:8:evicted_time 0
STAT items:8:outofmemory 0
STAT items:8:tailrepairs 0
STAT items:8:reclaimed 4980
STAT items:8:expired_unfetched 0
STAT items:8:evicted_unfetched 0
STAT items:9:number 22
STAT items:9:age 50
STAT items:9:evicted 0
STAT items:9:evicted_nonzero 0
STAT items:9:evicted_time 0
STAT items:9:outofmemory 0
STAT items:9:tailrepairs 0
STAT items:9:reclaimed 6601
STAT items:9:expired_unfetched 0
STAT items:9:evicted_unfetched 0
STAT items:10:number 20
STAT items:10:age 52
STAT items:10:evicted 0
STAT items:10:evicted_nonzero 0
STAT items:10:evicted_time 0
STAT items:10:outofmemory 0
STAT items:10:tailrepairs 0
STAT items:10:reclaimed 20550
STAT items:10:expired_unfetched 0
STAT items:10:evicted_unfetched 0
STAT items:11:number 26
STAT items:11:age 50
STAT items:11:evicted 0
STAT items:11:evicted_nonzero 0
STAT items:11:evicted_time 0
STAT items:11:outofmemory 0
STAT items:11:tailrepairs 0
STAT items:11:reclaimed 7857
STAT items:11:expired_unfetched 0
STAT items:11:evicted_unfetched 0
STAT items:12:number 13
STAT items:12:age 50
STAT items:12:evicted 0
STAT items:12:evicted_nonzero 0
STAT items:12:evicted_time 0
STAT items:12:outofmemory 0
STAT items:12:tailrepairs 0
STAT items:12:reclaimed 2656
STAT items:12:expired_unfetched 0
STAT items:12:evicted_unfetched 0
STAT items:13:number 9
STAT items:13:age 50
STAT items:13:evicted 0
STAT items:13:evicted_nonzero 0
STAT items:13:evicted_time 0
STAT items:13:outofmemory 0
STAT items:13:tailrepairs 0
STAT items:13:reclaimed 2737
STAT items:13:expired_unfetched 0
STAT items:13:evicted_unfetched 0
STAT items:14:number 17
STAT items:14:age 52
STAT items:14:evicted 0
STAT items:14:evicted_nonzero 0
STAT items:14:evicted_time 0
STAT items:14:outofmemory 0
STAT items:14:tailrepairs 0
STAT items:14:reclaimed 8091
STAT items:14:expired_unfetched 0
STAT items:14:evicted_unfetched 0
STAT items:15:number 10
STAT items:15:age 49
STAT items:15:evicted 0
STAT items:15:evicted_nonzero 0
STAT items:15:evicted_time 0
STAT items:15:outofmemory 0
STAT items:15:tailrepairs 0
STAT items:15:reclaimed 2836
STAT items:15:expired_unfetched 0
STAT items:15:evicted_unfetched 0
STAT items:16:number 8
STAT items:16:age 50
STAT items:16:evicted 0
STAT items:16:evicted_nonzero 0
STAT items:16:evicted_time 0
STAT items:16:outofmemory 0
STAT items:16:tailrepairs 0
STAT items:16:reclaimed 5097
STAT items:16:expired_unfetched 0
STAT items:16:evicted_unfetched 0
STAT items:17:number 5
STAT items:17:age 50
STAT items:17:evicted 0
STAT items:17:evicted_nonzero 0
STAT items:17:evicted_time 0
STAT items:17:outofmemory 0
STAT items:17:tailrepairs 0
STAT items:17:reclaimed 3297
STAT items:17:expired_unfetched 0
STAT items:17:evicted_unfetched 0
STAT items:18:number 1
STAT items:18:age 45
STAT items:18:evicted 0
STAT items:18:evicted_nonzero 0
STAT items:18:evicted_time 0
STAT items:18:outofmemory 0
STAT items:18:tailrepairs 0
STAT items:18:reclaimed 763
STAT items:18:expired_unfetched 0
STAT items:18:evicted_unfetched 0
STAT items:19:number 2
STAT items:19:age 119
STAT items:19:evicted 0
STAT items:19:evicted_nonzero 0
STAT items:19:evicted_time 0
STAT items:19:outofmemory 0
STAT items:19:tailrepairs 0
STAT items:19:reclaimed 786
STAT items:19:expired_unfetched 0
STAT items:19:evicted_unfetched 0
STAT items:20:number 2
STAT items:20:age 52
STAT items:20:evicted 0
STAT items:20:evicted_nonzero 0
STAT items:20:evicted_time 0
STAT items:20:outofmemory 0
STAT items:20:tailrepairs 0
STAT items:20:reclaimed 820
STAT items:20:expired_unfetched 0
STAT items:20:evicted_unfetched 0
STAT items:21:number 2
STAT items:21:age 592
STAT items:21:evicted 0
STAT items:21:evicted_nonzero 0
STAT items:21:evicted_time 0
STAT items:21:outofmemory 0
STAT items:21:tailrepairs 0
STAT items:21:reclaimed 420
STAT items:21:expired_unfetched 0
STAT items:21:evicted_unfetched 0
STAT items:22:number 2
STAT items:22:age 41
STAT items:22:evicted 0
STAT items:22:evicted_nonzero 0
STAT items:22:evicted_time 0
STAT items:22:outofmemory 0
STAT items:22:tailrepairs 0
STAT items:22:reclaimed 3975
STAT items:22:expired_unfetched 0
STAT items:22:evicted_unfetched 0
STAT items:23:number 2
STAT items:23:age 52
STAT items:23:evicted 0
STAT items:23:evicted_nonzero 0
STAT items:23:evicted_time 0
STAT items:23:outofmemory 0
STAT items:23:tailrepairs 0
STAT items:23:reclaimed 2398
STAT items:23:expired_unfetched 0
STAT items:23:evicted_unfetched 0
STAT items:24:number 1
STAT items:24:age 5
STAT items:24:evicted 0
STAT items:24:evicted_nonzero 0
STAT items:24:evicted_time 0
STAT items:24:outofmemory 0
STAT items:24:tailrepairs 0
STAT items:24:reclaimed 892
STAT items:24:expired_unfetched 0
STAT items:24:evicted_unfetched 0
STAT items:25:number 1
STAT items:25:age 5
STAT items:25:evicted 0
STAT items:25:evicted_nonzero 0
STAT items:25:evicted_time 0
STAT items:25:outofmemory 0
STAT items:25:tailrepairs 0
STAT items:25:reclaimed 158
STAT items:25:expired_unfetched 0
STAT items:25:evicted_unfetched 0
STAT items:26:number 1
STAT items:26:age 546
STAT items:26:evicted 0
STAT items:26:evicted_nonzero 0
STAT items:26:evicted_time 0
STAT items:26:outofmemory 0
STAT items:26:tailrepairs 0
STAT items:26:reclaimed 19
STAT items:26:expired_unfetched 0
STAT items:26:evicted_unfetched 0
STAT items:29:number 1
STAT items:29:age 688
STAT items:29:evicted 0
STAT items:29:evicted_nonzero 0
STAT items:29:evicted_time 0
STAT items:29:outofmemory 0
STAT items:29:tailrepairs 0
STAT items:29:reclaimed 3
STAT items:29:expired_unfetched 0
STAT items:29:evicted_unfetched 0
STAT items:32:number 1
STAT items:32:age 70054
STAT items:32:evicted 0
STAT items:32:evicted_nonzero 0
STAT items:32:evicted_time 0
STAT items:32:outofmemory 0
STAT items:32:tailrepairs 0
STAT items:32:reclaimed 0
STAT items:32:expired_unfetched 0
STAT items:32:evicted_unfetched 0

I am using

• Ubuntu 14.04
• PHP
• Silverstripe 3.5.0
• Zend_Cache (part of Silvertripe)
• Memcache version: 1.4.14

Below you can see some more stats.

server stats

slab stats

here is an interesting test I am running in PHP:

...
if(isset($_GET['setm'])) {
    foreach(array('11211', '11212', '11213', '11214') as $port) {
        echo "<h1>SETTING: $port</h1>";
        $memcache = new Memcache;
        $cacheAvailable = $memcache->connect('127.0.0.1', $port);
        if($cacheAvailable) {
            $memcache->set('test_memcache', 'set at: '.date('Y-m-d H:i:s'));
            echo "SET";
        } else {
            echo "NOT SET";
        }
    }
} elseif(isset($_GET['getm'])) {
    foreach(array('11211', '11212', '11213', '11214') as $port) {
        echo "<h1>GETTING: $port</h1>";
        $memcache = new Memcache;
        $cacheAvailable = $memcache->connect('127.0.0.1', $port);
        if($cacheAvailable) {
            $outcome = $memcache->get('test_memcache');
            echo $outcome;
        } else {
            echo "COULD NOT GET";
        }
    }
}
...

first of all I run: setm ... all four caches are usually set... sometimes one of them does not set!

Then I run getm:

GETTING: 11211
GETTING: 11212
set at: 2016-12-08 16:35:46
GETTING: 11213
set at: 2016-12-08 16:35:46
GETTING: 11214
set at: 2016-12-08 16:35:46

Now, the data from the website under heavy load immediately disappears, the data from the less busy website disappears after a minute or so, and the other data sticks around.

What is going on here?

I would really appreciate some pointers as memcache is new to me.

Answer 1

STAT cmd_flush 19612

The SilverStripe CMS has an internal cache cleaner (called clean()), which it calls 25ish places. This is implemented by running flush_all against memcached, which increments that counter and causes the entire cache to be lost.

The memcached daemon is working by design: memory is being reclaimed (see the 'reclaimed' stat under the items one there) after being asynchronously flushed by SilverStripe.

It's up to them to implement memcached in a way that doesn't do that.