Ah, I did a bit more reading on calling conventions. For cdecl, it said that sometimes non-POD data types >64bits (e.g. struct with 3 DWORDs) will be returned "by hidden parameter on the stack". I'm guessing this is what you mean.
I'm still confused why I need two args for my detour to work in Windows, and why having only one arg fails. Is it because my detour needs to make the hidden parameter explicit? For detours, do I need to pass thisptr explicitly? I don't see any other detours doing that...
I looked at some other functions, and I'm becoming more and more certain that the order of the arguments changes between the Windows and Linux versions. Unlike the other detours I've examined, thisptr for Windows is definitely not the first argument in Linux for UseHealingItems.
EDIT:
I compared Martijn's .so to the Linux disassembly. The args are (probably) passed correctly, so I don't think that's the problem anymore (although it's still weird that the argument order was rearranged...). But I think I see what's causing the problem now.
The calling convention used by UseHealingItems doesn't match anything I've seen before. It looks like the callee is allocating space on the stack for its own arguments (??) but I assume this is just gcc's way of optimizing things. But it gets weirder! After the callee clears its stack, it executes RETN 4 - but all the arguments are already cleared! And sure enough, if you follow it back to the caller, the caller is doing a SUB esp, 4 to undo the unnecessary RETN 4.
So here's my theory. The game gets detoured. I jump on the trampoline. When execution returns, the RETN 4 has corrupted the stack, and since the compiler didn't add a SUB esp, 4 to compensate, my detour's function epilogue is screwed (it probably ends up popping the return address into ebp or something like that)
Now, I can compensate for this with some dirty in-line assembly right after the function call. BUT the problem is that my detour's epilogue ends with RETN 0 - but the game is expecting UseHealingItems to RETN 4! So when the game tries to uncorrupt the stack, it will screw everything up again. The only thing I can think of to fix this is patching the extension after its built, so that it does RETN 4. But RETN 0 is a one-byte instruction, and RETN 4 has three bytes, but there's alignment padding that I can probably take advantage of for the extra 2 bytes.
...so, does any of that make any sense to anyone else, or do I just sound crazy? I remember reading that compilers are free to come up with their own non-standard calling conventions if a function is declared static, so is that a possible explanation for the odd behavior I'm seeing?
__________________
Last edited by dcx2; 07-31-2013 at 01:10.
Reason: Further investigation
|
Linear Mode
