How does the VM OS issue privileged Opcodes without being terminated by the Host OS/CPU?

Question

My understanding is that the host Operating System upon booting associates with and locks certain CPU Opcodes/instructions so that only the host OS may use them. The CPU architecture, additionally, supports this.

If a program offends and tries to use these instructions, the CPU will trap and revert control back to the OS which will terminate the program. Hence a program uses Syscalls() to do these things.

When you Virtualized an OS, it wouldn't issue syscalls(), it would simply execute the privileged instruction. It stands to reason it would then be terminated by the Host OS.

How are Opcodes virtualized?

Answer 1

There's another option besides termination, which is emulation and continue.

Some systems emulate misaligned loads & stores when the hardware doesn't support it but programs expect it; similar for floating point instructions & registers.  So, a host can emulate things to make it look like the hardware did execute that instruction.  Emulation means making changes to the program's state (cpu registers or memory as needed) and then resuming the program after the faulting instruction.

Similar mechanism for handling page faults: load, copy, or otherwise map the address and then continue the program — though for these, the faulting instruction is restarted once the page(s) are made available.

In those environments, it is critical that the privileged and missing instructions fault (take a hardware exception) so that the host can emulate with the proper semantics.  The hardware changes behavior for execution of privileged instructions based on the operating mode, whether user or supervisor (or other).  Taking an exception automatically switches to another mode, and resuming from the exception restores the original mode.

Hardware schemes can help with efficiency, for example, RISC V defines specific instructions and behaviors for multiple separate privilege levels, so that some privileged operations can be allowed without faulting to the next layer.

Answer 2

If a program offends and tries to use these instructions, the CPU will trap and revert control back to the OS

Yes, that's what the hardware does.

which will terminate the program.

Well, not necessarily. At this point it's completely up to the OS how to proceed. It's true that termination is usually the default behavior, but other options are possible.

For instance, under Unix-like operating systems, what really happens is that a signal is delivered to the process (e.g. SIGILL, SIGSEGV, SIGBUS). By default, this terminates the process (optionally with a core dump). But if the program previously installed a handler for the signal in question, then instead of terminating the process, the OS instead passes control to the signal handler (after switching back into unprivileged user mode). It can also make available to the handler whatever information about the fault was recorded by the CPU: address of faulting instruction, error codes, etc (on Unix this is passed in an optional third context argument to the handler). The handler can then do whatever it thinks is appropriate, such as emulating the privileged instruction and jumping back to the main line of execution.

---

I think it's a common misconception that it's necessary for security that an unprivileged process attempting a privileged operation must be terminated, as a sort of "death penalty" punishment. That's not really true. As far as security is concerned, all that's necessary is that the privileged operation doesn't take place. Whatever happens instead is more about usability.

Terminating the process is certainly a sensible default. The assumption is that a typical unprivileged program normally has no reason to execute a privileged instruction; the programmer or compiler ought to know that it's not going to work. The OS could just skip that instruction and let the process continue, and there'd be no security problem. But since the program is not behaving as the programmer presumably intended, letting it go on would simply let it get more and more confused. It may go on to do something that it has permission to do, but which the user doesn't want (output incorrect results, overwrite the user's data, or simply waste CPU and memory). So it makes sense to terminate it at once, to notify the user that something went wrong and avoid further malfunction. This is the behavior that gives the most value to the user.

But the programmer can override that assumption by installing a signal handler (or equivalent on other systems). This effectively tells the OS: "I am aware that I may be executing such instructions, and that they would trap; that's exactly what I want." That would be the situation for virtualization. Or, alternatively, "I don't actually intend to execute such instructions, but in case I do, I want to be able to recover, or to clean up before exiting." And if that's what the programmer wants, there's no reason for the OS to object.