[rhelgeby]

For some time I've been using static arrays to simulate objects. Now I'm moving towards dynamic structures such as ADT Arrays and ADT Tries, because it flexible and much easier to work with.

I'm using this for both small and large "objects". The result is a lot more handles being created and I'm worried I'll simply run out of free handles. This is not a memory leak, but maybe I use ADT Arrays in a way that wasn't originally anticipated (or any other data structure that creates a handle).

I assume the handle system is global, and from what I remember a handle is a 32 bit integer (don't remember if its unsigned or not). Some of the bits in the handle number specifies the handle type and possibly other meta data. If I understand the source correctly, the maximum number of handles is ~16,000.

About 16,000 handles total for all plugins seems a bit low if many of those plugins are using dynamic data structures (the new Zombie:Reloaded will do). At some point we'll reach the limit even though all handles are still in use (many small "objects"). Maybe not one plugin alone will create that many objects, but it's plausible that all of them in total may do so.

I was wondering how the handle system could be improved (hypothetically) to allow more handles per plugin?

My first guess would be to separate the handle value and its meta data, if it's possible without degrading performance too much.

[Powerlord]

Originally Posted by rhelgeby For some time I've been using static arrays to simulate objects. Now I'm moving towards dynamic structures such as ADT Arrays and ADT Tries, because it flexible and much easier to work with. I'm using this for both small and large "objects". The result is a lot more handles being created and I'm worried I'll simply run out of free handles. This is not a memory leak, but maybe I use ADT Arrays in a way that wasn't originally anticipated (or any other data structure that creates a handle).

This is why we really need structs in SourcePawn. They were one of the intended features for SourcePawn 2.0, but that project is on indefinite hold.

Having said that, most plugins don't use lots of Handles... most only use them for cvars and maybe a data structure here and there.

Originally Posted by rhelgeby I assume the handle system is global, and from what I remember a handle is a 32 bit integer (don't remember if its unsigned or not). Some of the bits in the handle number specifies the handle type and possibly other meta data. If I understand the source correctly, the maximum number of handles is ~16,000. About 16,000 handles total for all plugins seems a bit low if many of those plugins are using dynamic data structures (the new Zombie:Reloaded will do). At some point we'll reach the limit even though all handles are still in use (many small "objects"). Maybe not one plugin alone will create that many objects, but it's plausible that all of them in total may do so.

The Handle system, by its very nature, must be global. Otherwise, you wouldn't be able to pass Handles between extensions and plugins, or between plugins and other plugins.

The actual handle ID appears to be the lowest 16 bits of the Handle value... and the upper 16-bits are the Handle serial to make sure the Handle value hasn't been reassigned to a different Handle, much like how entity references work.

Now, having said that, HANDLESYS_MAX_HANDLES is (1<<14) (or 2^14 = 16,384). Its unclear why this value is used... and since it's used with > and not >=, bit (0-based) 14 is used only with bits 0-13 all set to 0... and bit 15 is never used.

It could be because C++ doesn't like large arrays and Handle indexes are the keys to an array of pointers that reference QHandle structs (which store info about the Handle's contents, including a pointer to any object the core / extension specified).

Originally Posted by rhelgeby I was wondering how the handle system could be improved (hypothetically) to allow more handles per plugin? My first guess would be to separate the handle value and its meta data, if it's possible without degrading performance too much.

Handle value and meta data are already separate. That's kinda the point of pointers.

[WildCard65]

Originally Posted by Powerlord most plugins don't use lots of Handles... most only use them for cvars and maybe a data structure here and there.

If that's true Powerlord, is what I'm doing a bit overkill? https://github.com/WildCard65/FF2-Al..._fortress_2.sp

Summarize what I'm doing with ADT Arrays and ADT Tries:
1 trie handle is the master handle containing everything related to it (incase of above link, let's say g_hBossCache), and it follows a keyvalues like structure and how I'm iterating it is with an ADT array placed inside the trie it contains the keys for and I grab the handle from the trie I want to iterate (unless I know exactly what keys are in the trie 100%, it'll be ommitted.), now for case above, they are a few exceptions to my keyNames array logic when it comes down to having non-handles and handles inside of a trie and one of the handles must be freed by plugin that created it, that's where "isSpecial" key comes in, it overrides the protection in the code I use to free all the handles (protection against handle leaking).

[asherkin]

You're over-thinking this, the limit is arbitrary.

https://bugs.alliedmods.net/show_bug.cgi?id=6065

[Powerlord]

Originally Posted by WildCard65 If that's true Powerlord, is what I'm doing a bit overkill? https://github.com/WildCard65/FF2-Al..._fortress_2.sp Summarize what I'm doing with ADT Arrays and ADT Tries: 1 trie handle is the master handle containing everything related to it (incase of above link, let's say g_hBossCache), and it follows a keyvalues like structure and how I'm iterating it is with an ADT array placed inside the trie it contains the keys for and I grab the handle from the trie I want to iterate (unless I know exactly what keys are in the trie 100%, it'll be ommitted.), now for case above, they are a few exceptions to my keyNames array logic when it comes down to having non-handles and handles inside of a trie and one of the handles must be freed by plugin that created it, that's where "isSpecial" key comes in, it overrides the protection in the code I use to free all the handles (protection against handle leaking).

I'll note that I haven't looked at the code.

If you're just using ADT stuff to make a keyvalues like structure, why not just use a keyvalues structure? KV isn't necessarily bad as long as you don't try to make gigantic structures from them.

I probably shouldn't be talking since I'm currently guilty of storing Handle values of adt structures inside a KeyValues, something I should really fix...

I mean, the server literally has thousands of KVs loaded into memory at any given time (every entity has one)...

[rhelgeby]

It might not be an issue, it's just that I'm a bit worried I'll hit a limit that isn't really necessary.

If it really becomes an issue I could instead rewrite my object library as an extension and create my own object references. I guess there already are object-frameworks I could just expose to SourceMod plugins.

[WildCard65]

Originally Posted by Powerlord I'll note that I haven't looked at the code. If you're just using ADT stuff to make a keyvalues like structure, why not just use a keyvalues structure? KV isn't necessarily bad as long as you don't try to make gigantic structures from them. I probably shouldn't be talking since I'm currently guilty of storing Handle values of adt structures inside a KeyValues, something I should really fix... I mean, the server literally has thousands of KVs loaded into memory at any given time (every entity has one)...

Have you seen my KV parser progress so far for FF2-Alpha?

[asherkin]

Originally Posted by rhelgeby It might not be an issue, it's just that I'm a bit worried I'll hit a limit that isn't really necessary. If it really becomes an issue I could instead rewrite my object library as an extension and create my own object references. I guess there already are object-frameworks I could just expose to SourceMod plugins.

If you have an issue with the limit, chime in on the bug I linked. We have no problem increasing it, hardware has moved on a lot since 2004 - the limit is just there for sanity to catch broken plugins.

[xf117]

I miss ES Python so much with classes, powerful std and additional libraries
But i guess if you writing something that big that requires classes, you are expected to write it in C++...
At least structures would be nice tho. I wonder how hard is it to implement something like that into existing language.

[rhelgeby]

Originally Posted by asherkin If you have an issue with the limit, chime in on the bug I linked. We have no problem increasing it, hardware has moved on a lot since 2004 - the limit is just there for sanity to catch broken plugins.

Sounds good. The optimal solution might be a limit per plugin (using compiler parameters or directives), if it's possible without rewriting the handle system. We'll get back to this on the bug report if it's an issue.