can't catch SIGTERM sent by upstart in golang

1.3k Views Asked by At

I have an upstart job running (some kind of TCP server). occasionally my process get restarted and I can see the next line in my syslog:

kernel: [2422352.460162] init: <job_name> main process (16545) killed by TERM signal

I don't understand the reason for this TERM signal by the kernel so I decided to catch the signal and print some memory and goroutines statistics before I terminate.
so now my code looks like this:

func main() {
   sigc := make(chan os.Signal, 1)
   signal.Notify(sigc, syscall.SIGTERM)
   go func() {
      s := <-sigc
      numOfGoRoutines := runtime. NumGoroutine()
      var stats runtime.MemStats
      runtime.ReadMemStats(&stats)
      log.Println("Got Signal ", s)
      log.Println("num of goroutines: ", numOfGoRoutines)
      log.Println("Memory Allocated: ",  stats.Alloc)
      os.Exit(1)
   }()
   someInfiniteLoopFunction() // this function defined in a different file.

now the weird thing is that my goroutine doesn't catch the TERM signal even when I manually stop the job.
even more interesting is the fact that if I add a Sleep for 100 seconds right after signal handler goroutine and stop the job during this sleep period then my goroutine DOES catch the TERM signal.
I'm absolutely clueless here and will appreciate any help.

1

There are 1 best solutions below

4
On

UPD. Sorry, I read it correctly now. Maybe your goroutine don't wait signal and execute os.Exit(). Try to add another chan.

func main() {
   sigc := make(chan os.Signal, 1)
   done := make(chan bool, 1)
   signal.Notify(sigc, syscall.SIGTERM)
   go func() {

      s := <-sigc
      numOfGoRoutines := runtime. NumGoroutine()
      var stats runtime.MemStats
      runtime.ReadMemStats(&stats)
      log.Println("Got Signal ", s)
      log.Println("num of goroutines: ", numOfGoRoutines)
      log.Println("Memory Allocated: ",  stats.Alloc)
      done <- true
      os.Exit(1)
   }()
   <-done
   someInfiniteLoopFunction() // this function defined in a different file.