[pRED*]

It looks like __usercall is IDA's name for a calling convention that it doesn't know.

Does this binary have link time code generation enabled?

If you disassemble your code you should see that the detour handler isn't marked as __usercall, and this is why it isn't working.

You could look at writing a __declspec(naked) detour handler to explicitly manage the two register based parameters?