Golang, processes and shared memory

Question

Today a friend of mine told me that Go programs can scale themselves on multiple CPU cores. I were quite surprised to hear that knowing that system task schedulers do not know anything about goroutines and hence can't run them on multiple cores.

I did some search and found out that Go programs can spawn multiple OS tasks to run them on different cores (the number is controlled by GOMAXPROCS environment variable). But as far as I know forking a process leads to complete copy of process data and different processes run in different address spaces.

So what about global variables in Go programs? Are they safe to use with multiple goroutines? Do they somehow synchronize between system processes? And if they do then how? I am mainly concerned about linux and freebsd implementations.

Answer 1

I figured it out! It's all in go sources.

There is a Linux system call that I were unaware of. It's called "clone". It is more flexible than fork and it allows a child process to live in its parent's address space.

Here is a short overview of the thread creation process.

First there is a newm function in src/runtime/proc.go. This function is responsible for creating a new working thread (or machine as it is called in comments).

// Create a new m. It will start off with a call to fn, or else the scheduler.
// fn needs to be static and not a heap allocated closure.
// May run with m.p==nil, so write barriers are not allowed.
//go:nowritebarrier
func newm(fn func(), _p_ *p) {

    // ... some code skipped ...

    newosproc(mp, unsafe.Pointer(mp.g0.stack.hi))
}

This function calls newosproc which is OS-specific. For Linux it can be found in src/runtime/os_linux.go. Here are relevant parts of that file:

var (
    // ...

    cloneFlags = _CLONE_VM | /* share memory */
        _CLONE_FS | /* share cwd, etc */
        _CLONE_FILES | /* share fd table */
        _CLONE_SIGHAND | /* share sig handler table */
        _CLONE_THREAD /* revisit - okay for now */
)

// May run with m.p==nil, so write barriers are not allowed.
//go:nowritebarrier
func newosproc(mp *m, stk unsafe.Pointer) {

    // ... some code skipped ...

    ret := clone(cloneFlags, /* ... other flags ... */)

    // ... code skipped
}

And the clone function is defined in architecture-specific files. For amd64 it is in src/runtime/sys_linux_amd64.s. It is the actual system call.

So Go programs do run in multiple OS threads which enables spanning across CPUs, but they use one shared address space.

Phew... I love Go.