Memory is not even getting freed (back to OS)

Question

I'm running a server written in Go and the RSS is very high and the memory is not even getting freed (back to OS).

I used pprof to check but it seems that there is no memory leak.

I also tried:

GODEBUG=madvdontneed=1 ./memorytest

Please tell me how to use madvdontneed.

OS: CentOS 7 (Linux)
Arch: amd64
Go version: 1.14.2

Code:

package main

import (
    "fmt"
    "os"
    "runtime"
    "time"
)

var garr []string

var chn chan int

func main() {
    chn = make(chan int, 1)

    go Alloc()

    <-chn
}

func Alloc() {
    for {
        arr := make([]string, 100000000)
        //copy(garr,arr)
        garr = arr
        print_heap_info()
        time.Sleep(5 * time.Second)
    }
}

func print_heap_info() {
    var m runtime.MemStats
    runtime.ReadMemStats(&m)
    fmt.Printf("env:%v, heapsys:%d,heapalloc:%d,heapidel:%d,heapreleased:%d,heapinuse:%d\n",
        os.Getenv("GODEBUG"), m.HeapSys, m.HeapAlloc, m.HeapIdle, m.HeapReleased, m.HeapInuse)
}

Output：

env:madvdontneed=1, heapsys:1677262848,heapalloc:1600074304,heapidel:76955648,heapreleased:76914688,heapinuse:1600307200
env:madvdontneed=1, heapsys:3287875584,heapalloc:3200081392,heapidel:87556096,heapreleased:87506944,heapinuse:3200319488
env:madvdontneed=1, heapsys:4898619392,heapalloc:4800086512,heapidel:98295808,heapreleased:98115584,heapinuse:4800323584
env:madvdontneed=1, heapsys:6509101056,heapalloc:6400090640,heapidel:108773376,heapreleased:108724224,heapinuse:6400327680
env:madvdontneed=1, heapsys:6509232128,heapalloc:4800086176,heapidel:1708908544,heapreleased:108724224,heapinuse:4800323584
env:madvdontneed=1, heapsys:6509002752,heapalloc:6400090304,heapidel:108675072,heapreleased:108560384,heapinuse:6400327680
env:madvdontneed=1, heapsys:6509199360,heapalloc:4800086712,heapidel:1708875776,heapreleased:108560384,heapinuse:4800323584
env:madvdontneed=1, heapsys:6509068288,heapalloc:6400090744,heapidel:108740608,heapreleased:108560384,heapinuse:6400327680
env:madvdontneed=1, heapsys:6509199360,heapalloc:4800086712,heapidel:1708875776,heapreleased:108560384,heapinuse:4800323584
env:madvdontneed=1, heapsys:6509068288,heapalloc:6400090840,heapidel:108740608,heapreleased:108462080,heapinuse:6400327680

Answer 1

Optimize your memory allocations to lower the peak.

See also: debug.FreeOSMemory()

FreeOSMemory forces a garbage collection followed by an attempt to return as much memory to the operating system as possible. (Even if this is not called, the runtime gradually returns memory to the operating system in a background task.)

---

Your formatted output is:

sys: 1599 MB alloc: 1525 MB idel:   73 MB released:   73 MB inuse: 1526 MB 
sys: 3135 MB alloc: 3051 MB idel:   83 MB released:   83 MB inuse: 3052 MB 
sys: 4671 MB alloc: 4577 MB idel:   93 MB released:   93 MB inuse: 4577 MB 
sys: 6207 MB alloc: 6103 MB idel:  103 MB released:  103 MB inuse: 6103 MB 
sys: 6207 MB alloc: 4577 MB idel: 1629 MB released:  103 MB inuse: 4577 MB 
sys: 6207 MB alloc: 6103 MB idel:  103 MB released:  103 MB inuse: 6103 MB 
sys: 6207 MB alloc: 4577 MB idel: 1629 MB released:  103 MB inuse: 4577 MB 
sys: 6207 MB alloc: 6103 MB idel:  103 MB released:  103 MB inuse: 6103 MB 
sys: 6207 MB alloc: 4577 MB idel: 1629 MB released:  103 MB inuse: 4577 MB 
sys: 6207 MB alloc: 6103 MB idel:  103 MB released:  103 MB inuse: 6103 MB

Which is (for the last line):
sys: bytes of heap memory obtained from the OS: 6207 MB
alloc: bytes of allocated heap objects: 6103 MB
idel: bytes in idle (unused) spans: 103 MB
released: bytes of physical memory returned to the OS: 103 MB
inuse: bytes in in-use spans: 6103 MB

---

It is not a memory leak. Output for a system with total 8GB RAM (system memory monitor):

command:

GODEBUG=madvdontneed=1  go run .

Output:

env: madvdontneed=1, sys: 1087 MB, alloc: 1024 MB, idel:   63 MB, released:   63 MB, inuse: 1024 MB
env: madvdontneed=1, sys: 2111 MB, alloc: 2048 MB, idel:   63 MB, released:   63 MB, inuse: 2048 MB
env: madvdontneed=1, sys: 3135 MB, alloc: 3072 MB, idel:   63 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB

Code:

package main

import (
    "fmt"
    "os"
    "runtime"
    "time"
)

func main() {
    for i := 0; i < 10; i++ {
        a = make([]byte, 1024*meg)
        var m runtime.MemStats
        runtime.ReadMemStats(&m)
        fmt.Printf("env: %v, sys: %4d MB, alloc: %4d MB, idel: %4d MB, released: %4d MB, inuse: %4d MB\n",
            os.Getenv("GODEBUG"), m.HeapSys/meg, m.HeapAlloc/meg, m.HeapIdle/meg, m.HeapReleased/meg, m.HeapInuse/meg)
        time.Sleep(1 * time.Second)
    }
}

var a []byte

const meg = 1024 * 1024

---

htop:

---

Output for vmstat 1 -S MB command, running two terminals:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0   1161   6369     25    595    0    0   702   452  276  567  6  3 91  1  0
 3  0   1160   6300     28    632    0    0 43432     0 6288 14013 13  5 82  1  0
 1  0   1160   6243     28    634    0    0   528     0 6342 13008 11  3 85  0  0
 1  0   1160   6177     28    634    0    0    56    56  979 2090  4  1 95  0  0
 2  0   1160   6110     28    634    0    0     0     0 1061 2664  5  2 94  0  0
 0  0   1160   6044     28    634    0    0    12    36  994 2233  4  1 94  0  0
 0  0   1160   3991     28    634    0    0    32   188 1074 1787  5  6 89  0  0
 2  0   1160   2120     28    634    0    0     0   136 1016 1634  4  5 90  0  0
 1  0   1160    973     28    633    0    0     0     4 1077 1660  4  5 92  0  0
 2  0   1160    143     25    390    0    0  1780   356 1849 2341  4  9 87  0  0
 0  9   1323    102      0    108    0  165 51964 169652 20277 21017  1 20 53 25  0
 1  5   1510     99      0    129    2  189 54376 193996 57829 52152  1 16 56 27  0
 1  5   1794     99      0    129    1  286 10068 293856 81160 59511  0 16 77  6  0
 4  5   2047    101      0     98    5  257 21236 263292 69923 65485  0 23 54 23  0
 1  2   1479   2867      0    217   43   23 233508 24380 24023 48536  4 27 47 22  0
 2  0   1452   2824      0    232   27    0 43960     0 8168 21085  4  5 90  1  0
 0  1   1443   2814      0    233    9    0 10956     0 3341 8468  5  2 93  0  0
 3  0   1425   2796      0    231   18    0 19688     0 5780 15490  4  3 92  1  0
 1  1   1420   2672     10    337    3    0 121628  1920 3292 7934  5  7 81  7  0
 0  0   1394   2646     10    338   25    0 27360     0 7975 21555  3  5 92  0  0
 0  1   1359   6856     10    339    1    0  2416     0 1035 2108  3  2 95  0  0
 0  0   1353   6847     10    348    4    0 13660     0 1696 3471  4  1 95  0  0

Output for first GODEBUG=madvdontneed=1 go run . command (auto killed):

env: madvdontneed=1, sys: 1087 MB, alloc: 1024 MB, idel:   63 MB, released:   63 MB, inuse: 1024 MB
env: madvdontneed=1, sys: 2111 MB, alloc: 2048 MB, idel:   63 MB, released:   63 MB, inuse: 2048 MB
env: madvdontneed=1, sys: 3135 MB, alloc: 3072 MB, idel:   63 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB
signal: killed

Output for second GODEBUG=madvdontneed=1 go run . command:

env: madvdontneed=1, sys: 1087 MB, alloc: 1024 MB, idel:   63 MB, released:   63 MB, inuse: 1024 MB
env: madvdontneed=1, sys: 2111 MB, alloc: 2048 MB, idel:   63 MB, released:   63 MB, inuse: 2048 MB
env: madvdontneed=1, sys: 3135 MB, alloc: 3072 MB, idel:   63 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   62 MB, inuse: 4096 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   62 MB, inuse: 3072 MB
env: madvdontneed=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   62 MB, inuse: 4096 MB

---

Code:

package main

import (
    "fmt"
    "os"
    "runtime"
    "time"
)

func main() {
    for i := 0; i < 10; i++ {
        a = make([]byte, 1024*meg)
        var m runtime.MemStats
        runtime.ReadMemStats(&m)
        fmt.Printf("env: %v, sys: %4d MB, alloc: %4d MB, idel: %4d MB, released: %4d MB, inuse: %4d MB\n\n",
            os.Getenv("GODEBUG"), m.HeapSys/meg, m.HeapAlloc/meg, m.HeapIdle/meg, m.HeapReleased/meg, m.HeapInuse/meg)
        time.Sleep(1 * time.Second)
    }
}

var a []byte

const meg = 1024 * 1024

Command:

GODEBUG=gctrace=1 go run .

Output:

gc 1 @0.008s 2%: 0.071+0.67+0.034 ms clock, 0.57+0.88/0.76/0.041+0.27 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 2 @0.014s 3%: 0.069+1.3+0.027 ms clock, 0.55+0.48/0.80/0.73+0.21 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 3 @0.029s 2%: 0.056+0.62+0.040 ms clock, 0.45+0.39/0.75/0.67+0.32 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 4 @0.045s 3%: 0.31+0.97+0.12 ms clock, 2.5+0.91/1.2/0.92+1.0 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 5 @0.060s 2%: 0.056+0.60+0.019 ms clock, 0.45+0.44/0.70/1.1+0.15 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 6 @0.071s 2%: 0.025+1.0+0.018 ms clock, 0.20+0.47/1.1/3.5+0.15 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
gc 7 @0.084s 2%: 0.12+0.88+0.042 ms clock, 0.97+0.77/1.2/0.88+0.33 ms cpu, 4->4->1 MB, 5 MB goal, 8 P
gc 8 @0.093s 2%: 0.039+0.83+0.028 ms clock, 0.31+0.38/0.66/1.2+0.22 ms cpu, 4->4->0 MB, 5 MB goal, 8 P

gc 1 @0.007s 3%: 0.013+1.7+0.005 ms clock, 0.11+0.86/2.0/1.2+0.042 ms cpu, 4->5->4 MB, 5 MB goal, 8 P

gc 1 @0.002s 5%: 0.024+2.0+0.042 ms clock, 0.19+0.25/1.5/1.3+0.34 ms cpu, 4->6->5 MB, 5 MB goal, 8 P
gc 2 @0.017s 3%: 0.014+4.6+0.044 ms clock, 0.11+0.13/3.3/1.9+0.35 ms cpu, 9->10->7 MB, 10 MB goal, 8 P
gc 3 @0.058s 2%: 0.030+6.7+0.036 ms clock, 0.24+0.12/5.3/1.8+0.29 ms cpu, 13->15->10 MB, 15 MB goal, 8 P
gc 4 @0.100s 2%: 0.034+5.6+0.015 ms clock, 0.27+0/7.2/0.65+0.12 ms cpu, 18->18->12 MB, 21 MB goal, 8 P
gc env: gctrace=1, sys: 1087 MB, alloc: 1024 MB, idel:   63 MB, released:   63 MB, inuse: 1024 MB

1 @0.024s 0%: 0.028+0.41+0.016 ms clock, 0.22+0.11/0.14/0.093+0.13 ms cpu, 1024->1024->1024 MB, 1025 MB goal, 8 P
env: gctrace=1, sys: 2111 MB, alloc: 2048 MB, idel:   63 MB, released:   63 MB, inuse: 2048 MB

gc 2 @1.049s 0%: 0.021+0.44+0.005 ms clock, 0.16+0.12/0.15/0.12+0.045 ms cpu, 2048->2048->2048 MB, 2049 MB goal, 8 P
env: gctrace=1, sys: 3135 MB, alloc: 3072 MB, idel:   63 MB, released:   63 MB, inuse: 3072 MB

env: gctrace=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB

gc 3 @3.096s 0%: 0.023+0.56+0.017 ms clock, 0.18+0.13/0.20/0.17+0.13 ms cpu, 4096->4096->2048 MB, 4097 MB goal, 8 P
env: gctrace=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB

env: gctrace=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB

gc 4 @5.619s 0%: 0.023+0.31+0.018 ms clock, 0.18+0.18/0.18/0.22+0.15 ms cpu, 4096->4096->2048 MB, 4097 MB goal, 8 P
env: gctrace=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB

env: gctrace=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB

gc 5 @8.141s 0%: 0.025+0.26+0.004 ms clock, 0.20+0.16/0.15/0.15+0.033 ms cpu, 4096->4096->2048 MB, 4097 MB goal, 8 P
env: gctrace=1, sys: 4159 MB, alloc: 3072 MB, idel: 1087 MB, released:   63 MB, inuse: 3072 MB

gc 6 @10.413s 0%: 0.028+0.51+0.013 ms clock, 0.22+0.23/0.28/0.29+0.11 ms cpu, 4096->4096->2048 MB, 4097 MB goal, 8 P
env: gctrace=1, sys: 4159 MB, alloc: 4096 MB, idel:   63 MB, released:   63 MB, inuse: 4096 MB

The format of this line is subject to change. Currently, it is:

    gc # @#s #%: #+#+# ms clock, #+#/#/#+# ms cpu, #->#-># MB, # MB goal, # P
where the fields are as follows:
    gc #        the GC number, incremented at each GC
    @#s         time in seconds since program start
    #%          percentage of time spent in GC since program start
    #+...+#     wall-clock/CPU times for the phases of the GC
    #->#-># MB  heap size at GC start, at GC end, and live heap
    # MB goal   goal heap size
    # P         number of processors used

---

Note:
go version go1.15.5 linux/amd64

---

See also:
Go 1.13 RSS keeps on increasing, suspected scavenging issue
https://github.com/golang/go/issues/36398
https://github.com/golang/go/issues/39295
https://go.googlesource.com/proposal/+/master/design/14951-soft-heap-limit.md
https://docs.google.com/document/d/1wmjrocXIWTr1JxU-3EQBI6BK6KgtiFArkG47XK73xIQ/edit#
https://blog.cloudflare.com/go-dont-collect-my-garbage/