I am writing a small tool in golang that reads in CSV file and passes each row to a channel. There is a receiver go-routine for every row. Somehow that is very slow - using profiling I found that the program is 68% on runtime.usleep. I have no 'sleeps' in my code.
Can someone point me to where this could come from, what it means and how to get rid of it?
(pprof) top10
28.98s of 30.12s total (96.22%)
Dropped 64 nodes (cum <= 0.15s)
Showing top 10 nodes out of 44 (cum >= 19.38s)
flat flat% sum% cum cum%
20.53s 68.16% 68.16% 20.53s 68.16% runtime.usleep
6.86s 22.78% 90.94% 6.86s 22.78% syscall.Syscall
0.30s 1% 91.93% 0.30s 1% runtime/internal/atomic.Cas64
0.28s 0.93% 92.86% 0.28s 0.93% runtime.mach_semaphore_signal
0.26s 0.86% 93.73% 19.15s 63.58% runtime.findrunnable
0.24s 0.8% 94.52% 0.24s 0.8% runtime.mach_semaphore_wait
0.21s 0.7% 95.22% 17.11s 56.81% runtime.runqgrab
0.13s 0.43% 95.65% 0.23s 0.76% runtime.runqempty
0.12s 0.4% 96.05% 4.17s 13.84% runtime.lock
0.05s 0.17% 96.22% 19.38s 64.34% runtime.schedule
(pprof)
relevant source code:
[...snip from main...]
var wg_p sync.WaitGroup
var wg_w sync.WaitGroup
for i := 0; i < workercount; i++ {
wg_p.Add(1)
go func(i int) {
processor(i, chn_read, chn_write)
wg_p.Done()
}(i)
}
wg_w.Add(1)
go func() {
writer(chn_write)
wg_w.Done()
}()
// read headers/1st line from csv
headers, _ := csvf.Read()
g_fieldnames = headers
chn_write <- g_fieldnames
// read 'n feed linewise
for {
chunk, _ := csvf.Read()
if chunk == nil {
close(chn_read)
break
}
chn_read <- chunk
}
[...snip end from main...]
and the go-routines
func processor(num_p int, chn_read chan []string, chn_write chan []string) {
for msg := range chn_read {
// snipped working on 'msg'
chn_write <- msg
}
}
func writer(chn_write chan []string) {
fout, _ := os.Create("./out.csv")
defer fout.Close()
w := csv.NewWriter(fout)
counter := 0
for msg := range chn_write {
w.Write(msg)
counter++
}
w.Flush()
}