Raised This Month: $12 Target: $400
 3% 

Solved Hooking a function with unusual call conventions?


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
nosoop
Veteran Member
Join Date: Aug 2014
Old 03-24-2018 , 03:08   Hooking a function with unusual call conventions?
Reply With Quote #1

Still doing NextBot stuff.

This time I'm attempting to port over a Source:Python plugin called PLRBots (a plugin that patches TFBots to support Payload Race) to an extension.

Right now I'm running into an issue attempting to call the original function from a hooked CTFGameRules::GetPayloadToPush. The author of the S:Py plugin had the same issue here.

Here's a disassembly view of the start of the function:



The demangled symbol suggests a sole integer argument, but the linked forum question (as well as the disassembly, I think) indicates that two arguments are provided (in addition to the implicit this). Here's a link to the S:Py equivalent of gamedata, the highlighted section suggesting it takes a pointer and an integer argument (a train watcher handle and a team respectively) on the Linux version.

I decided to hook it as a two-argument function. I seem to be able to correctly get the team argument (not sure if the handle is correct; I can work that out later), but passing the values into DETOUR_MEMBER_CALL causes the server to eventually crash.

Here's an abbreviated portion of the code:

Code:
CDetour *dt_CTFGameRules_GetPayloadToPush;

DETOUR_DECL_MEMBER2(CTFGameRules_GetPayloadToPush, void, CBaseHandle*, handle, int, team) {
    // prints "called CTFGameRules::GetPayloadToPush(3)", team index is correct
    rootconsole->ConsolePrint("%s(%d)", "called CTFGameRules::GetPayloadToPush", team);
    // crashes here sooner or later
    DETOUR_MEMBER_CALL(CTFGameRules_GetPayloadToPush)(handle, team);
}

bool CTFBotExtension::SDK_OnLoad(char *error, size_t maxlen, bool late) {
    if (!gameconfs->LoadGameConfigFile("tfbotext", &g_pGameConf, error, maxlen)) {
        return false;
    }
    
    CDetourManager::Init(g_pSM->GetScriptingEngine(), g_pGameConf);
    
    dt_CTFGameRules_GetPayloadToPush = DETOUR_CREATE_MEMBER(CTFGameRules_GetPayloadToPush, "CTFGameRules::GetPayloadToPush(int)");
    if (dt_CTFGameRules_GetPayloadToPush == nullptr) {
        snprintf(error, maxlen, "%s", "Failed to load CTFGameRules::GetPayloadToPush detour");
        return false;
    }
    
    dt_CTFGameRules_GetPayloadToPush->EnableDetour();
    return true;
}
I've also tried detouring as a struct with similar results:

Code:
struct payloadargs_t {
    CBaseHandle* handle;
    int team;
};

DETOUR_DECL_MEMBER1(CTFGameRules_GetPayloadToPush, void, payloadargs_t, args) {
    // prints "called CTFGameRules::GetPayloadToPush(3)", team index is correct
    rootconsole->ConsolePrint("%s(%d)", "called CTFGameRules::GetPayloadToPush", args.team);

    // crashes here sooner or later
    DETOUR_MEMBER_CALL(CTFGameRules_GetPayloadToPush)(args);
}
Any ideas on how to call the function correctly, as well as suggestions on how the arguments should be set up?

Last edited by nosoop; 03-25-2018 at 10:49.
nosoop is offline
psychonic

BAFFLED
Join Date: May 2008
Old 03-24-2018 , 08:41   Re: Hooking a function with unusual call conventions?
Reply With Quote #2

Going by memory for some of this. Hopefully it's correct.

For any function that returns a class/struct instance by value, MSVC compiles it as you say, with an added param that passing the value by ref. I believe GCC and Clang do the same, but only if the class is larger than the size of a pointer (which it isn't in this case). I believe there's still a return value, a pointer to the return object.

I think that last piece is the only part you're missing in your first try above.

Edit: that said, the compiler just might do the work for you, depending on what CDetour's macros create. You might be able to just do:
DETOUR_DECL_MEMBER1(CTFGameRules_GetPayloadTo Push, CBaseHandle, int, team)

Last edited by psychonic; 03-24-2018 at 08:43.
psychonic is offline
nosoop
Veteran Member
Join Date: Aug 2014
Old 03-24-2018 , 11:01   Re: Hooking a function with unusual call conventions?
Reply With Quote #3

Thanks for the insight and suggestions; that's a lot of black magic. As far as MSVC goes, that does seem to line up with what I've seen in the Python plugin.

If it's of any relevance, I'm using GCC 5.4 (Linux). I also tried Clang, though I think I ended up with similar results.

I've tried the following; still pretty new to C++ so it's possible I'm doing something incorrectly:
Code:
// also crashes 
DETOUR_DECL_MEMBER2(CTFGameRules_GetPayloadToPush, void*, CBaseHandle*, handle, int, team) {
    DETOUR_MEMBER_CALL(CTFGameRules_GetPayloadToPush)(handle, team);
    return handle;
}

// no crash though the bots don't seem to recognize the cart
DETOUR_DECL_MEMBER1(CTFGameRules_GetPayloadToPush, void*, void*, args) {
    return DETOUR_MEMBER_CALL(CTFGameRules_GetPayloadToPush)(args);
}
Haven't been able to check your specific suggestion as I'm not sure how to prevent the incomplete type error the CBaseHandle declaration is giving me (how would I pull in a full definition of CBaseHandle?).

My original attempt was based off of these hooks in the S:Py plugin, which suggests that it doesn't use the return on Linux (unless the server plugin itself is doing similar magic).

I wouldn't mind setting up a barebones extension with just that detour if a hands-on thing would help (though I'd have to do that later in the day, about to hit the bed).
__________________
I do TF2, TF2 servers, and TF2 plugins.
I don't do DMs over Discord -- PM me on the forums regarding inquiries.
AlliedModders Releases / Github / TF2 Server / Donate (BTC / BCH / coffee)
nosoop is offline
hmmmmm
Great Tester of Whatever
Join Date: Mar 2017
Location: ...
Old 03-24-2018 , 23:49   Re: Hooking a function with unusual call conventions?
Reply With Quote #4

For the CBaseHandle stuff try including gamehelpers (by unccommenting gamehelpers thing in smsdk_config.h)
Not sure if there is a better way for that but that's easiest way I've found to get CBaseEntity/CBaseHandle types

Last edited by hmmmmm; 03-24-2018 at 23:49.
hmmmmm is offline
nosoop
Veteran Member
Join Date: Aug 2014
Old 03-25-2018 , 10:49   Re: Hooking a function with unusual call conventions?
Reply With Quote #5

I already included gamehelpers; the compiler mentions that there's only a forward declaration of CBaseHandle from the IGameHelpers header; can't use the class directly as a (non-pointer) function argument / return type.

Pulled in a stub from the SDK's basehandle include, and it looks like it's working as predicted.
Thanks for the help, duck!

I've set up a repository with just the relevant code here for future reference. This commit for the solution.

Edit: Looks like making MMS a requirement for the extension works too (had to pull in FindEntityByClassname and found a redeclaration error).
__________________
I do TF2, TF2 servers, and TF2 plugins.
I don't do DMs over Discord -- PM me on the forums regarding inquiries.
AlliedModders Releases / Github / TF2 Server / Donate (BTC / BCH / coffee)

Last edited by nosoop; 03-26-2018 at 04:32.
nosoop is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 13:31.


Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Theme made by Freecode