golang profile runtime.usleep at 70%

1.5k Views Asked by At

I am writing a small tool in golang that reads in CSV file and passes each row to a channel. There is a receiver go-routine for every row. Somehow that is very slow - using profiling I found that the program is 68% on runtime.usleep. I have no 'sleeps' in my code.

Can someone point me to where this could come from, what it means and how to get rid of it?

(pprof) top10
28.98s of 30.12s total (96.22%)
Dropped 64 nodes (cum <= 0.15s)
Showing top 10 nodes out of 44 (cum >= 19.38s)
  flat  flat%   sum%        cum   cum%
20.53s 68.16% 68.16%     20.53s 68.16%  runtime.usleep
 6.86s 22.78% 90.94%      6.86s 22.78%  syscall.Syscall
 0.30s     1% 91.93%      0.30s     1%  runtime/internal/atomic.Cas64
 0.28s  0.93% 92.86%      0.28s  0.93%  runtime.mach_semaphore_signal
 0.26s  0.86% 93.73%     19.15s 63.58%  runtime.findrunnable
 0.24s   0.8% 94.52%      0.24s   0.8%  runtime.mach_semaphore_wait
 0.21s   0.7% 95.22%     17.11s 56.81%  runtime.runqgrab
 0.13s  0.43% 95.65%      0.23s  0.76%  runtime.runqempty
 0.12s   0.4% 96.05%      4.17s 13.84%  runtime.lock
 0.05s  0.17% 96.22%     19.38s 64.34%  runtime.schedule
(pprof)

relevant source code:

[...snip from main...]
var wg_p sync.WaitGroup
var wg_w sync.WaitGroup

for i := 0; i < workercount; i++ {
    wg_p.Add(1)
    go func(i int) {
        processor(i, chn_read, chn_write)
        wg_p.Done()
    }(i)
}

wg_w.Add(1)
go func() {
    writer(chn_write)
    wg_w.Done()
}()

// read headers/1st line from csv
headers, _ := csvf.Read()
g_fieldnames = headers
chn_write <- g_fieldnames

// read 'n feed linewise
for {
    chunk, _ := csvf.Read()
    if chunk == nil {
        close(chn_read)
        break
    }
    chn_read <- chunk
}
[...snip end from main...]

and the go-routines

func processor(num_p int, chn_read chan []string, chn_write chan []string) {
    for msg := range chn_read {
        // snipped working on 'msg'
        chn_write <- msg
    }
}

func writer(chn_write chan []string) {
    fout, _ := os.Create("./out.csv")
    defer fout.Close()
    w := csv.NewWriter(fout)
    counter := 0
    for msg := range chn_write {
        w.Write(msg)
        counter++
    }
    w.Flush()
}
0

There are 0 best solutions below