[LambdaLambda]

Hey,

im building plugin, which going to be handling ~200k of records, and I've got problem with handling all that.

My first idea was to build variable thats going to handle max caches of 10k (because at this stage, we thoght it's going to be enough). I've decided to do this that way, so server wont have to query database all the time, and all records are available in case they're needed.

Then, to sort all these, I decided not to query DB again, but i made function that does that. And it works, thats fine. However... We've found 10k won't be enough, and here's my problem.

I've found that when I'm trying to compile plugin with variable that would have to handle 200000 different records, its crashing during compiling. 100000 does fine.

Do you guys think, better would be to separate these between two variables, or let them to stay in database, and call them?

What i would like to get, is to make as less queries as it's possible. Let's say there are 10 servers using the same database. 40 players max on each, which means 400. Around 1/4 of them would finish the stage during the same minute - database goes crazy.

[Powerlord]

If you find yourself doing a bunch of inserts/updates/replaces at the same time, rather than doing a bunch of separate database queries, do a bunch in a threaded transaction.

That is SQL_CreateTransaction, SQL_AddQuery (once per query), and then at the end SQL_ExecuteTransaction.

Edit: Technically, you can also do these for SELECTs as well, they would appear in the success callback. Might be a good idea to pass in useful stuff like the userid/client serial as the data (which also appears as an array in the transaction success).

[m_bNightstalker]

ADT Arrays are sort-, ship- & swapable and it has no limit.

Originally Posted by Powerlord If you find yourself doing a bunch of inserts/updates/replaces at the same time, rather than doing a bunch of separate database queries, do a bunch in a threaded transaction. That is SQL_CreateTransaction, SQL_AddQuery (once per query), and then at the end SQL_ExecuteTransaction. Edit: Technically, you can also do these for SELECTs as well, they would appear in the success callback. Might be a good idea to pass in useful stuff like the userid/client serial as the data (which also appears as an array in the transaction success).

When I understand him correctly, he needs to do that often not all at once. So I would prefer to just update the database and modify the cache local.

[Potato Uno]

So using a bunch of adt_arrays

PHP Code:

new Handle:MyData = CreateArray(args);
... 


to store all this data isn't an option? You don't need to declare the size of the array prior (other than having to declare the block size if you're holding strings), so you can shove in as many entries in there as you want (up until whatever memory limit sourcemod gives to a plugin before NOPE'ing out).

If you want some insurance, some backup, or want to save the data to disk, you can spin off an infinite timer that runs once every N seconds (say, 10-15) that will then dump everything into a database (and for that, read whatever Powerlord has written).

[LambdaLambda]

I bet my terminology sucks. Im sorry for that.

What I've done:

I declared the size:

PHP Code:

#define MAX_CACHE 500000 


After that, I used it with creation of new variable:

PHP Code:

new g_cache[MAX_TYPES][MAX_CACHE][RecordCache]; 


And now, i'm doing one query, on map start, that is sorted and putted into the variable one by one, from 0 to MAX on MAX_CACHE area.

And why it's so important for me to use it that way; by that, I'm declaring player's rank position. On player connect, the cache is checked for his record - if there is one, he get's ID (MAX_CACHE number). So thanks for that, I don't have to do any additional stuff, I don't have to create new functions that are going to be checking his record comparatively to others, but i simply print out the cache ID, that i would have to use anyway, to check any other data that it includes.

Originally Posted by m_bNightstalker When I understand him correctly, he needs to do that often not all at once. So I would prefer to just update the database and modify the cache local.

Correct.


I've never been using proper arrays in SourcePawn, so I'm bit confused. Could someone give me quick example?

[Potato Uno]

I'm in no condition to give examples of sourcepawn as I haven't played with adt_arrays a whole lot (just only a little bit but not much). However, maybe this would be useful?

https://sm.alliedmods.net/api/index....ad=file&id=35&

[LambdaLambda]

hmm, will google some examples. Thank you for your time. : )

[m_bNightstalker]

Oh you are trying to modify zipcores timer...but so far I know the latest version already works with ADT arrays. Why do you need the old structure?

[LambdaLambda]

No, im not. I'm doing my own version. Zipcore did not use sorting, he used to check players rank everytime he needed that, using way I'm doing it now, I don't have to do that, so it would be more optimized.

[m_bNightstalker]

Originally Posted by LambdaLambda No, im not. I'm doing my own version. Zipcore did not use sorting, he used to check players rank everytime he needed that, using way I'm doing it now, I don't have to do that, so it would be more optimized.

I think you misunderstood what I'm trying to say. You have posted code from an older version of the timer, thats why I thought you "just" modify it.

Anyway, you should use ADT Arrays, so you need no MAX_CACHE and it's compiling faster.

You can sort those arrays to, thats not a problem. I had the same problem some days ago: https://forums.alliedmods.net/showthread.php?t=255581
Asherkin told me to use SortADTArrayCustom and it's working great.