It happens on custom training job with tpu_v2 in us-central1. I followed "launch web terminal" link under training debugging in custom training job UI, but got the following message.
I should have the necessary permissions as I started the custom training job, through vertex ai pipeline as part of kubeflow pipeline. My docker image should also have bash as its entry point calls bash.
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f3b932d7a7c m=0 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: g 0: unknown pc 0x7f3b932d7a7c
stack: frame={sp:0x7fff78eca340, fp:0x0} stack=[0x7fff786cb8c0,0x7fff78eca8d0)
0x00007fff78eca240: 0x0000000000000000 0x0000000000000000
0x00007fff78eca250: 0x0000000000000000 0x0000000000000000
0x00007fff78eca260: 0x0000000000000000 0x0000000000000000
...
0x00007fff78eca370: 0x00007fff78eca4be 0x00007fff78eca4bf
0x00007fff78eca380: 0x0000000000000000 0x00007f3b932d5759
0x00007fff78eca390: 0x00000000007fff00 0x0000000000000000
0x00007fff78eca3a0: 0x00000000003d0f00 0x00007f3b6bea8910
0x00007fff78eca3b0: 0x00007f3b6bea8910 0x00007f3b6bea8910
0x00007fff78eca3c0: 0x0000000000000000 0x71c8cb3b369b8900
0x00007fff78eca3d0: 0x00007f3b9323e740 0x0000000000000006
0x00007fff78eca3e0: 0x0000564c9f37f7db 0x00007fff78eca710
0x00007fff78eca3f0: 0x0000564c9fd99420 0x00007f3b93283476
0x00007fff78eca400: 0x00007f3b9345be90 0x00007f3b932697f3
0x00007fff78eca410: 0x0000000000000020 0x0000000000000001
0x00007fff78eca420: 0x00007f3b6bea8640 0x00007f3b932d65c4
0x00007fff78eca430: 0x00007fff78eca468 0x00007fff78eca650
goroutine 1 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:463 fp=0xc000132780 sp=0xc000132778 pc=0x564c9e777d20
runtime.main()
/usr/local/go/src/runtime/proc.go:170 +0x6d fp=0xc0001327e0 sp=0xc000132780 pc=0x564c9e748c2d
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001327e8 sp=0xc0001327e0 pc=0x564c9e779fc1
rax 0x0
rbx 0x7f3b9323e740
rcx 0x7f3b932d7a7c
rdx 0x6
rdi 0x6
rsi 0x6
rbp 0x6
rsp 0x7fff78eca340
r8 0x7fff78eca410
r9 0x7fffffff
r10 0x8
r11 0x246
r12 0x6
r13 0x16
r14 0x564c9fd99420
r15 0x1
rip 0x7f3b932d7a7c
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
[Finished... Terminado]