Tail call stack

Question

I'm having trouble understanding the Stack manipulation needed in order to implement Tail call in assembly language.

When we have a Tail call to function We basically want to override the current Activation Frame with the Activation Frame of the called function.

I'm Having trouble with understanding the process that is needed to take place in order to accomplish this switch of Activation Frames.

How does the RSP changes in order to be in the right address and how to I keep the RBP of the originial calling function from changing?

Answer 1

You just restore the the point that RSP is pointing at a return address (e.g. pop any registers you saved earlier, such as RBP), until the state of stack memory is exactly like on entry to this function.

Therefore a jmp to another function will work as if your caller had called that function instead.

(Except you can put different values in arg-passing registers, and in arg-passing space on the stack if there is any.)

In the simplest case, the entire function body is just jmp foo, like for https://godbolt.org/z/v57MG6fqW

int foo(int);

int bar(int x){
   return foo(x);
}

Using that Godbolt link, you can add stuff like volatile int a = 1; to see how a C compiler uses stack space in a function that eventually tail-calls. Compile with -mno-red-zone and/or -fno-omit-frame-pointer as well as -O3 to see more of a function epilogue.

In general, you run a normal function epilogue just like if you were going to return, and then you replace call/ret with jmp. The trick is being able to delay a call to this point, after you've cleaned up the function's stack frame. Any args should already be computed and in arg-passing registers or stack locations.

Of course, a call/ret wouldn't truly be legal at this point on x86-64 because the stack alignment is wrong: RSP % 16 == 8 on function entry, rather than 0 before a call. Similarly for Windows x64 shadow space.