Why doesn't Perl's `$shm->remove` from `IPC::SharedMem` remove the shared memory segment?

Question

I wrote a simple web server that uses IPC::SharedMem (IPC::SharedMem->new(IPC_PRIVATE, $size, S_IRWXU)) and forks off processes to handle client requests. When the server exited cleanly (and killed all client processes before), the shared memory isn't removed, although my program end with these lines:

if (parse_options()) {
#...
    print "$me: removing Shared Memory\n"
        if ($verbose > 1);
    $shm->remove;
    my $ret = $shm->remove;
    print "$me: result of removing Shared Memory: $ret\n"
} else {
    exit(1);
}

Actually my process ends with these messages:

...
child_create[20166]: child exits with 0
_child_handler: got 20166, have 20166
_child_handler: child PID 20166 exited with 0
REST-server: no longer accepting connections on http://localhost:37709/:
REST-server: removing Shared Memory
REST-server: result of removing Shared Memory: 0 but true

So the last child process 20166 has been waited on, and the call is executed. Still there is shared memory left over (easily recognizable by user and size):

> ipcs

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 16646144   windl      700        4          0
0x00000000 16711681   windl      700        4          0
0x00000000 8880146    root       700        4          0

------ Semaphore Arrays --------
key        semid      owner      perms      nsems

(The one being owned by root is a different version running as root). As can be seen the nattach is zero, so it's not used any more, and ipcrm has no problem removing those (I wrote a little script to do it when testing my Perl program):

> ./clean-shm.sh
+ pcrm -m 16646144 -m 16711681

The manual says for the remove method:

Remove the shared memory segment from the system or mark it as removed as long as any processes are still attached to it.

So what's wrong? I'm running on Linux x86_64 (SLES 12, kernel 4.12.14-122.176-default), and the host is in VMware (virtualized).