AlliedModders - [Solved] Hooking a function with unusual call conventions?

AlliedModders (https://forums.alliedmods.net/index.php)

- Coding MM:S Plugins & SM Extensions (https://forums.alliedmods.net/forumdisplay.php?f=75)

- - Solved Hooking a function with unusual call conventions? (https://forums.alliedmods.net/showthread.php?t=306280)

Hooking a function with unusual call conventions?

Still doing NextBot stuff.

This time I'm attempting to port over a Source:Python plugin called PLRBots (a plugin that patches TFBots to support Payload Race) to an extension.

Right now I'm running into an issue attempting to call the original function from a hooked CTFGameRules::GetPayloadToPush. The author of the S:Py plugin had the same issue here.

Here's a disassembly view of the start of the function:

https://i.imgur.com/ojajc8Z.png

The demangled symbol suggests a sole integer argument, but the linked forum question (as well as the disassembly, I think) indicates that two arguments are provided (in addition to the implicit this). Here's a link to the S:Py equivalent of gamedata, the highlighted section suggesting it takes a pointer and an integer argument (a train watcher handle and a team respectively) on the Linux version.

I decided to hook it as a two-argument function. I seem to be able to correctly get the team argument (not sure if the handle is correct; I can work that out later), but passing the values into DETOUR_MEMBER_CALL causes the server to eventually crash.

Here's an abbreviated portion of the code:

Code:

CDetour *dt_CTFGameRules_GetPayloadToPush;



DETOUR_DECL_MEMBER2(CTFGameRules_GetPayloadToPush, void, CBaseHandle*, handle, int, team) {

    // prints "called CTFGameRules::GetPayloadToPush(3)", team index is correct

    rootconsole->ConsolePrint("%s(%d)", "called CTFGameRules::GetPayloadToPush", team);

    // crashes here sooner or later

    DETOUR_MEMBER_CALL(CTFGameRules_GetPayloadToPush)(handle, team);

}



bool CTFBotExtension::SDK_OnLoad(char *error, size_t maxlen, bool late) {

    if (!gameconfs->LoadGameConfigFile("tfbotext", &g_pGameConf, error, maxlen)) {

        return false;

    }

    

    CDetourManager::Init(g_pSM->GetScriptingEngine(), g_pGameConf);

    

    dt_CTFGameRules_GetPayloadToPush = DETOUR_CREATE_MEMBER(CTFGameRules_GetPayloadToPush, "CTFGameRules::GetPayloadToPush(int)");

    if (dt_CTFGameRules_GetPayloadToPush == nullptr) {

        snprintf(error, maxlen, "%s", "Failed to load CTFGameRules::GetPayloadToPush detour");

        return false;

    }

    

    dt_CTFGameRules_GetPayloadToPush->EnableDetour();

    return true;

}

I've also tried detouring as a struct with similar results:

Code:

struct payloadargs_t {

    CBaseHandle* handle;

    int team;

};



DETOUR_DECL_MEMBER1(CTFGameRules_GetPayloadToPush, void, payloadargs_t, args) {

    // prints "called CTFGameRules::GetPayloadToPush(3)", team index is correct

    rootconsole->ConsolePrint("%s(%d)", "called CTFGameRules::GetPayloadToPush", args.team);



    // crashes here sooner or later

    DETOUR_MEMBER_CALL(CTFGameRules_GetPayloadToPush)(args);

}

Any ideas on how to call the function correctly, as well as suggestions on how the arguments should be set up?

Re: Hooking a function with unusual call conventions?

Going by memory for some of this. Hopefully it's correct.

For any function that returns a class/struct instance by value, MSVC compiles it as you say, with an added param that passing the value by ref. I believe GCC and Clang do the same, but only if the class is larger than the size of a pointer (which it isn't in this case). I believe there's still a return value, a pointer to the return object.

I think that last piece is the only part you're missing in your first try above.

Edit: that said, the compiler just might do the work for you, depending on what CDetour's macros create. You might be able to just do:
DETOUR_DECL_MEMBER1(CTFGameRules_GetPayloadTo Push, CBaseHandle, int, team)

Re: Hooking a function with unusual call conventions?

Thanks for the insight and suggestions; that's a lot of black magic. As far as MSVC goes, that does seem to line up with what I've seen in the Python plugin.

If it's of any relevance, I'm using GCC 5.4 (Linux). I also tried Clang, though I think I ended up with similar results.

I've tried the following; still pretty new to C++ so it's possible I'm doing something incorrectly:

Code:

// also crashes 

DETOUR_DECL_MEMBER2(CTFGameRules_GetPayloadToPush, void*, CBaseHandle*, handle, int, team) {

    DETOUR_MEMBER_CALL(CTFGameRules_GetPayloadToPush)(handle, team);

    return handle;

}



// no crash though the bots don't seem to recognize the cart

DETOUR_DECL_MEMBER1(CTFGameRules_GetPayloadToPush, void*, void*, args) {

    return DETOUR_MEMBER_CALL(CTFGameRules_GetPayloadToPush)(args);

}

Haven't been able to check your specific suggestion as I'm not sure how to prevent the incomplete type error the CBaseHandle declaration is giving me (how would I pull in a full definition of CBaseHandle?).

My original attempt was based off of these hooks in the S:Py plugin, which suggests that it doesn't use the return on Linux (unless the server plugin itself is doing similar magic).

I wouldn't mind setting up a barebones extension with just that detour if a hands-on thing would help (though I'd have to do that later in the day, about to hit the bed).

Re: Hooking a function with unusual call conventions?

For the CBaseHandle stuff try including gamehelpers (by unccommenting gamehelpers thing in smsdk_config.h)
Not sure if there is a better way for that but that's easiest way I've found to get CBaseEntity/CBaseHandle types

Re: Hooking a function with unusual call conventions?

I already included gamehelpers; the compiler mentions that there's only a forward declaration of CBaseHandle from the IGameHelpers header; can't use the class directly as a (non-pointer) function argument / return type.

Pulled in a stub from the SDK's basehandle include, and it looks like it's working as predicted.
Thanks for the help, duck!

I've set up a repository with just the relevant code here for future reference. This commit for the solution.

Edit: Looks like making MMS a requirement for the extension works too (had to pull in FindEntityByClassname and found a redeclaration error).