[Potato Uno]

So I'm about to write a plugin that is pending to read off about 5-6 MB of VDF files, using adt_arrays to store the data and the keyvalues API available to parse the files. Not all the data needs to live in memory all at once, but I'd rather load it up into RAM now than read the files and parse it while the server is running with players (thus causing huge var spikes). The parsing is to happen during the server boot up (OnPluginStart), so I don't need to worry about optimization or I/O time or CPU usage.

My question is:

1. What's the memory limit that sourcepawn gives to a plugin before automatically unloading it?

2. How big can adt_arrays really get? Is it actually "unlimited" or is there some hard-coded limit to their overall array size?

3. Do I actually need to ever close the handles of these adt_arrays if I want to consistently use them all the time? (I believe SM will automatically close these handles when unloading the plugin or when killing the server?)

4. If I set a block size of 200 for each array and then shove 5 strings of length 20, does it still use 1000 bytes of data?

5. So that I have some ability to gauge my memory usage, how can I check how much memory a plugin is using?

I'd rather waste several [hundred] megabytes of RAM than to do I/O mid-game, since the former only is a price that is paid on server boot up (when nobody is there) and the latter is a price paid while people are in the server.

In other words, minimize I/O time as much as possible mid-game, is my main goal. (Unless if there's any additional problems that come with the server using a lot of RAM - if so let me know that too.)

Thanks!

[Mathias.]

1. not sure the default but you can decide to up the limit with #pragma dynamic

2. I dont think there is a limit but I can't confirm

3. You dont need, sourcemod will do it, but if your not going to use your adt array and let the plugin load you better CloseHandle to free the memory

4. It will use the rest of the memory, so yes.

You should not care too much about memory, use what you need except if you have leaks that a different story. For example create an atd array do you stuff and assign a new atd array to that handle without closing it before will create a memory leak.

Other options is to load the files at certain point of time in-game like round start, people will mostlikely not notice it and it won't affect a critical game play. Also you claim that you do not need all the information of the vdf files, you can create a replica into the memory but only with the sections you need (if you know what you want to do already)

[Potato Uno]

https://forums.alliedmods.net/showthread.php?t=223553

Shit I just realized this might be a problem.

Thank you for your responses by the way.

[asherkin]

Why not just keep using the KeyValues Handle rather than duplicating the data?

Sure, if you're only using a small percentage of what's in the file or need to do CPU-heavy manipulation to get it in a usable format, then it makes sense to parse and copy to working structures, but it doesn't sound like you are from your past posts on the subject.

[Potato Uno]

Originally Posted by asherkin Why not just keep using the KeyValues Handle rather than duplicating the data? Sure, if you're only using a small percentage of what's in the file or need to do CPU-heavy manipulation to get it in a usable format, then it makes sense to parse and copy to working structures, but it doesn't sound like you are from your past posts on the subject.

I do like the idea of keeping the KeyValues handle and then writing a function to look up data every time (saves the boot time and bigRAM usage), but I have to ask this question:

Wouldn't doing that cost more CPU time (and during the middle of a game no less) because it has to traverse down the key value file every single time? I was having the idea that putting everything into adt_arrays at the start and then doing a bunch of lookups (GetArrayCell/GetArrayString/GetArrayArray) in the array during the game since the lookups should barely cost any CPU time at all.

However, the idea of just reusing the KV handle sounds like an interesting one that I'll probably end up trying (depending on the performance overhead it causes).

The server isn't ran on potato hardware (i7-3770, no virtual machine) but I still like to keep CPU time at a minimum in sourcemod plugins since they run on the same thread as the main game thread (different story if they ran on a separate thread from the main thread).

[asherkin]

The KV API is extremely cheap as long as you use it properly (although the SourcePawn wrapping makes this a little harder), it's just a big nested tree storing each value using the type guessed at parse time, the underlying file isn't touched after parsing.

As long as you're careful to traverse it in a sane order and only unwinding when necessary rather than jumping all over the place, you really shouldn't have any trouble even with huge files (which 5MB isn't really near).

EDIT: You can change your own thread title by going into the "Advanced" editor for the first post and changing the "Title" field.

[Potato Uno]

I'm going to try it out tomorrow with the VDF files I have. Thank you for all the help!

[friagram]

Not sure what you are doing, but when you start getting megabytes of data, sql becomes a better idea, esp if you have a local sql implementation, or could benefit from sharing the data.

Most of the time you will know what you need ahead of time, and can query/cache it.
If you really don't know and can't have a delay, then a lookup with something like keyvalues could do (or sqlite?). I would recommend against keyvalues because as they get huge, they get stupid hard to manage. You also don't benefit from indexing and caching like with sql...

[Potato Uno]

SQL can definitely be an option, and I can readily interconvert the data between keyvalues and local databases, but won't that incur a higher I/O overhead during the server run since it needs to open the database every time that data needs to be accessed?

Is there perhaps a way to store a database entirely in memory? (It's read-only.)

[Potato Uno]

https://forums.alliedmods.net/showth...ghlight=Sqlite

Unless if things have changed since then, SQL local databases sound like a huge nope in performance.

Keyvalues are apparently faster. I'll have to deal with whatever pain in the ass management they give me for the speed benefit.