Windows x64 stack frame ABI shenanigans

23 Views Asked by At

so I am making a toy compiler to assembly and my program is failing at the boundary of my generated code and C code. This is what the assembly looks like:

(Pseudo-code from my compiler)
def test(x: int) -> int:
    return magic_func(magic_func(x, x), magic_func(x, x))
(Actual output assembly:)
test:
push rcx
mov rdx, rcx
call magic_func
mov rdx, rax
pop rcx
push rdx
mov rdx, rcx
call magic_func
mov rcx, rax
pop rdx
call magic_func
ret

The C code looks like this:

int magic_func(int x, int y) {
    printf("magic_func(%d, %d)\n", x, y);
    return x + y * 2;
}
extern int test(int);
int main() {
    printf("%d\n", test(44));
    return 0;
}

And the magic_func function specifically like this: (from objdump)

push   rbp
mov    rbp,rsp
sub    rsp,0x20
mov    DWORD PTR [rbp+0x10],ecx
mov    DWORD PTR [rbp+0x18],edx
mov    eax,DWORD PTR [rbp+0x18]
mov    r8d,eax
mov    edx,DWORD PTR [rbp+0x10]
lea    rcx,[rip+0x0]        # 1e <magic_func+0x1e>
call   23 <magic_func+0x23>
mov    eax,DWORD PTR [rbp+0x18]
lea    edx,[rax+rax*1]
mov    eax,DWORD PTR [rbp+0x10]
add    eax,edx
add    rsp,0x20
pop    rbp
ret

Now, I've found out with GDB that the segmentation fault occurs at the ret instruction of my function (test), but always at the third (final) invocation. The return address that was supposed to be saved on the stack is rewritten with some garbage. I'm sure this is some Windows stack shenanigans with alignment or some red zones or me not allocating enough space but I can't figure it out. Somebody with Windows calling convention experience who could give me a tip?

0

There are 0 best solutions below