Tarantool Cartridge "Memory is highly fragmented" errors

Question

I'm running a cluster with enough RAM per node, but nevertheless it's plagued by frequent "Memory is highly fragmented" errors. When looking at dashboard, it says only ~2.3-2.4 gigs out of 6.3 are used. Looking at Slabs it also sees low quota utilization, but items and arena are near or above 99%

> box.slab.info()
---
- items_size: 1286313040
  items_used_ratio: 99.56%
  quota_size: 6500000768
  quota_used_ratio: 40.27%
  arena_used_ratio: 96.2%
  items_used: 1280601176
  quota_used: 2617245696
  arena_size: 2617245696
  arena_used: 2516986968
...

I tried adjusting Vinyl memory and Vinyl cache, as well as changing memtx_min_tuple_size from 16 bits to 64, but it didn't affect the ratios.

Could please someone explain how to increase items and arena size? or maybe there are other ways to fix this?

With default value of fragmentation_threshold_critical = 0.85 this cluster should run with ~3 gigs of ram and still have some room

items_used_ratio = items_used / items_size
quota_used_ratio = quota_used / quota_size
arena_used_ratio = arena_used / arena_size

UPDATE: adding slab stats as requested

box.slab.stats()
---
- - mem_free: 16400
    mem_used: 260224
    item_count: 4066
    item_size: 64
    slab_count: 17
    slab_size: 16384
  - mem_free: 6072
    mem_used: 10200
    item_count: 75
    item_size: 136
    slab_count: 1
    slab_size: 16384
  - mem_free: 16128
    mem_used: 52574976
    item_count: 365104
    item_size: 144
    slab_count: 3232
    slab_size: 16384
  - mem_free: 156216
    mem_used: 286979496
    item_count: 1888023
    item_size: 152
    slab_count: 17646
    slab_size: 16384
  - mem_free: 2943632
    mem_used: 423399040
    item_count: 2646244
    item_size: 160
    slab_count: 26201
    slab_size: 16384
  - mem_free: 913912
    mem_used: 405490008
    item_count: 2413631
    item_size: 168
    slab_count: 12445
    slab_size: 32768
  - mem_free: 862448
    mem_used: 288143152
    item_count: 1637177
    item_size: 176
    slab_count: 8850
    slab_size: 32768
  - mem_free: 484712
    mem_used: 170306168
    item_count: 925577
    item_size: 184
    slab_count: 5230
    slab_size: 32768
  - mem_free: 53680
    mem_used: 44456448
    item_count: 231544
    item_size: 192
    slab_count: 1363
    slab_size: 32768
  - mem_free: 33208
    mem_used: 13617000
    item_count: 68085
    item_size: 200
    slab_count: 418
    slab_size: 32768
  - mem_free: 25792
    mem_used: 6276816
    item_count: 30177
    item_size: 208
    slab_count: 193
    slab_size: 32768
  - mem_free: 22144
    mem_used: 2361744
    item_count: 10934
    item_size: 216
    slab_count: 73
    slab_size: 32768
  - mem_free: 27104
    mem_used: 887264
    item_count: 3961
    item_size: 224
    slab_count: 28
    slab_size: 32768
  - mem_free: 28504
    mem_used: 396024
    item_count: 1707
    item_size: 232
    slab_count: 13
    slab_size: 32768
  - mem_free: 29376
    mem_used: 166560
    item_count: 694
    item_size: 240
    slab_count: 6
    slab_size: 32768
  - mem_free: 5216
    mem_used: 92752
    item_count: 374
    item_size: 248
    slab_count: 3
    slab_size: 32768
  - mem_free: 11296
    mem_used: 54016
    item_count: 211
    item_size: 256
    slab_count: 2
    slab_size: 32768
  - mem_free: 19720
    mem_used: 12936
    item_count: 49
    item_size: 264
    slab_count: 1
    slab_size: 32768
  - mem_free: 26416
    mem_used: 692016
    item_count: 2218
    item_size: 312
    slab_count: 22
    slab_size: 32768
  - mem_free: 30000
    mem_used: 35424
    item_count: 108
    item_size: 328
    slab_count: 1
    slab_size: 65536
  - mem_free: 2816
    mem_used: 62608
    item_count: 182
    item_size: 344
    slab_count: 1
    slab_size: 65536
  - mem_free: 33024
    mem_used: 32400
    item_count: 90
    item_size: 360
    slab_count: 1
    slab_size: 65536
  - mem_free: 36472
    mem_used: 28952
    item_count: 77
    item_size: 376
    slab_count: 1
    slab_size: 65536
  - mem_free: 54448
    mem_used: 10976
    item_count: 28
    item_size: 392
    slab_count: 1
    slab_size: 65536
  - mem_free: 12688
    mem_used: 249008
    item_count: 394
    item_size: 632
    slab_count: 4
    slab_size: 65536
  - mem_free: 123328
    mem_used: 7632
    item_count: 6
    item_size: 1272
    slab_count: 1
    slab_size: 131072
  - mem_free: 259416
    mem_used: 2616
    item_count: 1
    item_size: 2616
    slab_count: 1
    slab_size: 262144
  - mem_free: 11962320
    mem_used: 870891520
    item_count: 53155
    item_size: 16384
    slab_count: 421
    slab_size: 2097152
...

Answer 1

Could please someone explain how to increase items and arena size?

They are increased automatically. 99% of items_used_ratio and arena_used_ratio is fine so long as quota_used_ratio is low (40% in your case).

Actually it was a bug in the code that raises the warning, it was fixed in cartridge version 2.7.9.