I have many goroutines running for a very long time working with RabbitMQ. They receive data, accumulate it into sync.Map, and then every minute flush them to the database.
Before, when data was received, I was writing it straightaway to the database, but after some time it increased database' load dramatically, so I resorted to the solution of doing the flush every so and often.
Since then the memory usage of the app has been going up until it exhausts it and crashes. I didn't run a pprof yet, considering the new approach to be the cause, because when I revert it back the issue disappears.
I have many (a couple of thousand of goroutines) writing to sync.Map under integer key a map[string]interface{}. The question is why the memory usage increases and not gets GCed? This is the code for the flush method:
func flushOrdersToDatabase() error {
ordersRep := order.NewRepository(structs.Gorm)
log.Printf("Writing %d updated orders to database", lenSyncMap(&structs.Orders))
structs.Orders.Range(func(existingOrderID, columnsToUpdate interface{}) bool {
data := columnsToUpdate.(map[string]interface{})
errUpdateOrder := ordersRep.UpdateOrder(nil, existingOrderID.(uint), data)
if errUpdateOrder != nil {
log.Printf("Error updating data for order with ID %d: %s",
existingOrderID, errUpdateOrder.Error())
}
structs.Orders.Delete(existingOrderID)
return true
})
log.Printf("Flush of orders to database completed")
return nil
}
I have found that it's recommended to store a pointer to data instead of actual map, but haven't tried it yet.
This is how I populate the map:
queuedColumns, loaded := structs.Orders.LoadOrStore(existingOrder.ID, columnsToUpdate)
if loaded {
mergedResult := mergeMaps(queuedColumns.(map[string]interface{}), columnsToUpdate)
structs.Orders.Store(existingOrder.ID, mergedResult)
}
and this is the mergeMaps function just in case:
func mergeMaps(left map[string]interface{}, right map[string]interface{}) map[string]interface{} {
result := make(map[string]interface{})
for k, v := range left {
if _, ok := left[k]; ok {
result[k] = v
}
}
for k, v := range right {
if _, ok := right[k]; ok {
result[k] = v
}
}
return result
}