segment limit check in AMD 64-bit mode

Question

I am writing my own OS for 64bit processors and I am stuck with the problem of general protection. My OS will not rely on page fault to implement user space protection mechanism, so I found there is a way to do it with segment limit checking:

This presentation from VMWare

http://download3.vmware.com/vmworld/2005/pac346.pdf

on page 20 says:

• Initial AMD64 architecture did not include segmentation in 64-bit mode

  • Segmentation also missing from EMT64T

  How do we protect the VMM ?

  • 64-bit guest support requires additional hardware assitance
  • Segment limit checks available in 64-bit mode on newer AMD processors

Now, I have the newer AMD processor model and my question is how do I achieve limit segment limit check on AMD processor in 64-bit (long) mode ? I have downloaded the Sep 2011 version (lastest) of developer's manual and I can't find how to do this in any place, please help.

Answer 1

I think they're probably talking about the Long Mode Segment Limit Enable bit (LMSLE bit 13) in the "Extended Feature Enable Register" (EFER) in Volume 2 3.1.7 pg. 55. It is describe in a little more detail in "4.12.2 Data Limit Checks in 64-bit Mode" on page 114. Note that EFER is a model-specific register (more in "6.2.5 Accessing Model-Specific Register" pg. 156 also in volume 2).

Answer 2

Segmentation is an old and very slow way to implement memory protection. Even when it came out, nobody used it because it was too slow - Intel invented this but didn't actually talk to OS vendors to see what they wanted first. You really need to use page faulting like other modern operating systems.

Answer 3

Although this doesn't answer the question, this is related to removal of segment limit checks in 64 bit mode supposedly making protection of hypervisor trap handlers 'impossible' without hardware virtualisation, which is a discussion people may expect to see seeing this question title. 'breaking some existing implementations' I agree with, but not 'impossible'.

The initial version of x86-64 (AMD64) did not allow for a software-only full virtualization due to the lack of segmentation support in long mode, which made the protection of the hypervisor's memory impossible, in particular, the protection of the trap handler that runs in the guest kernel address space.

I see this all over but I'm not convinced. You don't need segmentation to protect hypervisor trap handlers or the IDT. You can do it with paging.

Make a certain virtual address range in the SPT always map the hypervisor trap handlers, and the IDT at the virtual address it is at on the host. The IDT requires 1 4KiB page. These pages in the SPT are made supervisor, meaning the guest kernel in ring 1 cannot write to them because it will cause a trap right into the IDT which is mapped into the guest. Now ring 0, the code can be executed. When the guest does read/write to those reserved virtual address ranges, it needs to be unaware that it is reserved, i.e. the hypervisor driver silently redirects the accesses with certain CR3s to a clean page. A read/write from a ring 1 guest kernel will cause a GPF, and the CR2 will contain the address that was attempted to be written to. Usually, the page is faulted in and then the RIP pushed is of the instruction to be reattempted, but it can't do this in this case. It needs to decode and perform the read/write itself at the RIP in the trap frame to a new host physical page by translating it using a special internal table purely for those reserved regions built when the guest modifies the PTEs, and then increase the RIP.