so I am making a toy compiler to assembly and my program is failing at the boundary of my generated code and C code. This is what the assembly looks like:
(Pseudo-code from my compiler)
def test(x: int) -> int:
return magic_func(magic_func(x, x), magic_func(x, x))
(Actual output assembly:)
test:
push rcx
mov rdx, rcx
call magic_func
mov rdx, rax
pop rcx
push rdx
mov rdx, rcx
call magic_func
mov rcx, rax
pop rdx
call magic_func
ret
The C code looks like this:
int magic_func(int x, int y) {
printf("magic_func(%d, %d)\n", x, y);
return x + y * 2;
}
extern int test(int);
int main() {
printf("%d\n", test(44));
return 0;
}
And the magic_func function specifically like this: (from objdump)
push rbp
mov rbp,rsp
sub rsp,0x20
mov DWORD PTR [rbp+0x10],ecx
mov DWORD PTR [rbp+0x18],edx
mov eax,DWORD PTR [rbp+0x18]
mov r8d,eax
mov edx,DWORD PTR [rbp+0x10]
lea rcx,[rip+0x0] # 1e <magic_func+0x1e>
call 23 <magic_func+0x23>
mov eax,DWORD PTR [rbp+0x18]
lea edx,[rax+rax*1]
mov eax,DWORD PTR [rbp+0x10]
add eax,edx
add rsp,0x20
pop rbp
ret
Now, I've found out with GDB that the segmentation fault occurs at the ret instruction of my function (test), but always at the third (final) invocation. The return address that was supposed to be saved on the stack is rewritten with some garbage. I'm sure this is some Windows stack shenanigans with alignment or some red zones or me not allocating enough space but I can't figure it out. Somebody with Windows calling convention experience who could give me a tip?