Changing segments to Ring3(user) from Ring0(kernel) results in segmentation fault

Question

I am writing a toy operating system. I am trying to implement memory protection for an operating system. All I want to do is create protection for the kernel from user space programs. I want to do this purely with segmentation and not paging.

Here is the GDT:

gdt_start:
    
    dd 0x0 ; 4 byte
    dd 0x0 ; 4 byte


gdt_code: 
    dw 0xfff7    ; segment length, bits 0-15
    dw 0x0       ; segment base, bits 0-15
    db 0x0       ; seg2ment base, bits 16-23
    db 10011010b ; flags (8 bits)
    db 11001111b ; flags (4 bits) + segment length, bits 16-19
    db 0x0       ; segment base, bits 24-31


gdt_data:
    dw 0xfff7
    dw 0x0
    db 0x0
    db 10010010b
    db 11001111b
    db 0x0

U_code: 
    dw 0xfff7    ; segment length, bits 0-15
    dw 0x0008       ; segment base, bits 0-15
    db 0x0       ; seg2ment base, bits 16-23
    db 11111010b ; flags (8 bits)
    db 11001111b ; flags (4 bits) + segment length, bits 16-19
    db 0x0       ; segment base, bits 24-31


U_data:
    dw 0xfff7
    dw 0x0008
    db 0x0
    db 11110010b
    db 11001111b
    db 0x0
gdt_end:

U_data and U_code are going to be the user space (ring 3). When I am in kernel space (ring 0) and try switching data segments by executing:

mov ax, 0x20 
mov ds, ax
mov ss, ax
mov es, ax
mov fs, ax
mov gs, ax

I get a segmentation error (13).

What am I doing wrong? Any guidance would be greatly appreciated.

Answer 1

mov ss, ax

You are loading ss with a data segment whose DPL is 3, while your CPL is 0. That's not allowed. Your stack always has to be at the same privilege level as you. (Also, it wouldn't make any sense to load ss without loading esp at the same time.)

The switch to the user's stack needs to happen simultaneously with the transition to CPL 3. The main ways to do this are:

• RETF or IRET. When the destination segment is less privileged, it will pop ss:esp from the (old) stack as well as cs:eip.

• A hardware task switch, in which case ss:esp is loaded from the TSS along with all the other registers.

Call gates also perform a stack switch, but you can only use them to call more privileged code, not the other way around.

Also, as noted in comments, your base and limit fields seem to be mixed up. As it stands your user segments have a base of 0x00000008 and all your segments have a limit of 0xffff7000. Note that "bits 0-15" are the least significant bits. So if it U_data is supposed to be the upper half of memory, then you would want

    dw 0xffff
    dw 0x0000
    db 0x0
    db 11110010b
    db 11000111b  ; note the low nibble is now 7
    db 0x80

Note that this is probably still not what you want, as there may or may not be any memory there (e.g. if your machine has less than 2 GB), and there may also be I/O mapped into that region which the user should not have access to. So you'd really need to inspect the memory map to decide where to put the user's segments.