Addressing variables (or, what is ML64 generating?)

Question

I have an ASM file written for X64. It provides a leaf function. The file was assembled with MASM64/ML64.

The C-pseduo code with a signature is:

int GenerateBlock(byte* ptr, size_t size, unsigned int safety)
{
    if (ptr == NUL) return 0; /*FAIL*/
    ...
}

Here's the same portion of ASM code:

;; RCX (in): byte* buffer
;; RDX (in): size_t bsize
;; R8d (in): unsigned int safety
;; RAX (out): bool, success (1), failure (0)

ASM_GenerateBlock PROC buffer:QWORD,bsize:QWORD,safety:DWORD
    LOCAL val:QWORD         ;; local variables
    MWSIZE EQU 8            ;; machine word size

            ;; Validate pointer
    cmp     buffer, 0
    je      ASM_GenerateBlock_Cleanup
            ;; Cleanup will set RAX to 0 and return
    ...
ENDP

When I step the call, it appears fastcall is being used, which is consistent with the docs. The first two args are showing up in RCX and RDX, which is also consistent.

But a test case with a NULL pointer is producing unexpected results. Here's the test case that's being used:

ASM_GenerateBlock(NULL /*ptr*/, 64 /*size*/, 20 /*safety*/);

When I step the code, RCX appears to be buffer (its NULL), RDX appears to be bsize (its 0x40), but the compare cmp buffer, 0 is occurring against a value that unknown to me. From the immediate window:

buffer
0x000000013f82bcd0
*(__int64*)buffer
0x000000013f62aba8
bsize
0x000000013f14871b
*(__int64*)bsize
0xccccccc348c48348

13f82bcd0 looks roughly like an instruction pointer address (EIP is 13F50D268). Its does not appear to be similar to ESP or EBP.

I have a few questions...

• What addressing mode is ML64 using for the variable buffer?
• Where is the value of the variable buffer coming from?
• Why is ML64 not using ECX for the variable buffer?
• How can I fix this?

---

The same code, shortened to 32-bits, assembles and executes fine. However, ML puts buffer and bsize on the stack and addresses them relative to EBP.

I also tried changing to cmp QWORD PTR buffer, 0, but it did not help.

---

Answer 1

From the disassembly in the final screenshot,

cmp  buffer, 0

is assembling to

cmp  qword ptr [buffer], 0   # memory operand.  rip-relative?  or stack-relative?  Not enough insn bytes for an absolute 32bit address

instead of

cmp  RCX, 0

So the assembly syntax you're using still declares buffer as a symbol or memory offset or something, not as an alias for a register. And yes, the x86-64 Windows ABI uses a register calling convention (unfortunately a different one from Linux). I guess it's similar to the 32bit fastcall ABI. Agner Fog has a doc explaining the various calling conventions for 32 and 64bit OSes.

---

Note that cmp with immediate zero is almost always a worse choice than test rcx, rcx. Shorter insn encoding, and still macro-fuses with a following jcc on Intel and AMD.

Answer 2

How can I fix this?

I can't answer some of the questions, but I know how to fix it. Below, and underlying requirement is the same source code work for both X86 (MASM/ML) and X64 (MASM64/ML64) with minimal changes.

Here was the original C function signature:

int GenerateBlock(byte* ptr, size_t size, unsigned int safety)

Under X86 MASM, relative addressing is used, and the ASM code would look as follows:

;; Base relative (in): byte* buffer
;; Base relative (in): size_t bsize
;; Base relative (in): unsigned int safety

ASM_GenerateBlock PROC buffer:DWORD,bsize:DWORD,safety:DWORD
    LOCAL val:DWORD         ;; local variables
    MWSIZE EQU 4            ;; machine word size

            ;; Validate pointer
    cmp     buffer, 0
    je      MSC_ASM_GenerateBlock_Cleanup    
    ...

    ;; Write byte to buffer from AL
    mov     BYTE PTR [buffer], al
    inc     buffer
    ...

For X64 with fastcall, a few minor hacks are required:

;; RCX (in): byte* buffer
;; RDX (in): size_t bsize
;; R8d (in): unsigned int safety

ASM_GenerateBlock PROC bufferX:QWORD,bsizeX:QWORD,safetyX:DWORD
    LOCAL val:QWORD         ;; local variables
    MWSIZE EQU 8            ;; machine word size

            ;; Fastcall workaround
    buffer EQU ecx
    bsize  EQU edx
    safety EQU r8d

            ;; Validate pointer
    cmp     buffer, 0
    je      MSC_ASM_GenerateBlock_Cleanup    
    ...

            ;; Write byte to buffer from AL
    mov     BYTE PTR [buffer], al
    inc     buffer
    ...

Above, two fixes occurred. First were the names of the variables in the procedure prototype. Buffer was changed to bufferX, etc. Second EQU was used like a C-language #define to equate buffer to ecx.