Raised This Month: $209 Target: $400
 52% 

[TF2] TF2 Item DB (replaces tf2itemsinfo)


Post New Thread Reply   
 
Thread Tools Display Modes
Author
bottiger
AlliedModders Donor
Join Date: Dec 2010
Plugin ID:
4497
Plugin Version:
0.93
Plugin Category:
Technical/Development
Plugin Game:
Team Fortress 2
Plugin Dependencies:
    Servers with this Plugin:
    496 
    Plugin Description:
    Old 01-22-2015 , 01:33   [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #1

    Want to help test out a new beta version of the plugin with more features? Click Here and please let us know if it works!

    This is a library that will allow you to query items or search for them with a specified criteria. For example you can check which slot an item uses or get all items in the melee slot for snipers.

    This plugin is meant to replace tf2itemsinfo which has memory leaks and other problems that cannot be fixed and will eventually become permanently broken due to how long it takes to run and that it uses almost the maximum number of handles.

    Data is stored in sqlite. A python script is used to update the database. I decided to do it this way because it is a bad idea to put a 10+ second loading time into your server especially when there is time limit for plugins now and is much faster in python. It is recommended to run this after every update. If you do not have python on your server, you can always run it on your computer and upload it to your server.

    We need people to start porting plugins to use tf2idb.

    I have tried to make the functions close to tf2itemsinfo but unfortunately I could not make them all the same because they were too inefficient, especially for SQLite.

    With the TF2IDB_FindItemCustom function, you can create arbitrary SQL queries which should be much faster and much more flexible than the finditem function in tf2itemsinfo.

    How to Install - Lazy Method

    You can use the pre-made database here but it is not recommended because I will not be updating it. You should follow the additional instructions below so you can make the database yourself after each update.
    • Put tf2idb.smx into your plugins folder.
    • Unzip the sample-db-date.zip file and put the sq3 file into your sourcemod/data/sqlite folder.


    How to Install - Additional steps to updating your own database
    • Install Python from python.org. If you are using Linux, it is probably already installed. Type python to check.
    • Unzip tf2idb.zip anywhere.
    • Follow the instructions below.

    You need to first edit tf2idb.py and point it to your TF2 server folder. It will look like this.

    Code:
    TF_FOLDER = r'C:/tf2server/tf/'
    ITEMS_GAME = TF_FOLDER + 'scripts/items/items_game.txt'
    DB_FILE = TF_FOLDER + 'addons/sourcemod/data/sqlite/tf2idb.sq3'
    Normally you will only need to change TF_FOLDER, but if you put items_game.txt or your sqlite folder in a different place, then you will need to change the other 2 lines.

    Then run tf2idb.py to create the database. If you need to update the database, run tf2idb.py again.


    Code:
    // If a native returns bool and it is not obvious from the name what it is, then the bool represents whether the native succeeded.
    // Natives returning a handle are ADT arrays. You must close them when you are done with them.
    
    native bool:TF2IDB_IsValidItemID(id);
    native bool:TF2IDB_GetItemName(id, String:string[], length);
    native bool:TF2IDB_GetItemClass(id, String:string[], length);
    native bool:TF2IDB_GetItemSlotName(id, String:string[], length);
    native TF2ItemSlot:TF2IDB_GetItemSlot(id);
    native bool:TF2IDB_GetItemQualityName(id, String:string[], length);
    native TF2ItemQuality:TF2IDB_GetItemQuality(id);
    native bool:TF2IDB_GetItemLevels(id, &min, &max);
    native TF2IDB_GetItemAttributes(id, aid[TF2IDB_MAX_ATTRIBUTES], Float:values[TF2IDB_MAX_ATTRIBUTES]);
    native Handle:TF2IDB_GetItemEquipRegions(id);
    native bool:TF2IDB_DoRegionsConflict(const String:region1[], const String:region2[]);
    native Handle:TF2IDB_ListParticles();
    native Handle:TF2IDB_FindItemCustom(const String:query[]);
    native bool:TF2IDB_ItemHasAttribute(id, aid);
    Some useful examples.

    Code:
    // If you are creating items, you will want to filter out items with string attributes like this.
    TF2IDB_FindItemCustom("SELECT id FROM tf2idb_item WHERE has_string_attribute=0");
    
    // get all medieval and melee items
    TF2IDB_FindItemCustom("SELECT a.id FROM tf2idb_item a JOIN tf2idb_item_attributes b ON a.id=b.id WHERE has_string_attribute=0 AND (attribute=2029 OR slot='melee') GROUP BY a.id");
    
    // get all heavy melee items
    TF2IDB_FindItemCustom("SELECT a.id FROM tf2idb_item a JOIN tf2idb_class b ON a.id=b.id WHERE b.class='heavy' AND slot='melee' AND has_string_attribute=0");
    Table definitions

    Code:
    ('CREATE TABLE "tf2idb_class" ("id" INTEGER NOT NULL , "class" TEXT NOT NULL , PRIMARY KEY ("id", "class"))')
    ('CREATE TABLE "tf2idb_item_attributes" ("id" INTEGER NOT NULL , "attribute" INTEGER NOT NULL , "value" TEXT NOT NULL, PRIMARY KEY ("id", "attribute") )')
    ('CREATE TABLE "tf2idb_item" ('
    	'"id" INTEGER PRIMARY KEY  NOT NULL ,'
    	'"name" TEXT NOT NULL,'
    	'"item_name" TEXT NOT NULL,'
    	'"class" TEXT NOT NULL,'
    	'"slot" TEXT,'
    	'"quality" TEXT NOT NULL,'
    	'"tool_type" TEXT,'
    	'"min_ilevel" INTEGER,'
    	'"max_ilevel" INTEGER,'
    	'"baseitem" INTEGER,'
    	'"holiday_restriction" TEXT,'
    	'"has_string_attribute" INTEGER'
    	')'
    )
    ('CREATE TABLE "tf2idb_particles" ("id" INTEGER PRIMARY KEY  NOT NULL , "name" TEXT NOT NULL )')
    ('CREATE TABLE "tf2idb_equip_conflicts" ("name" TEXT NOT NULL , "region" TEXT NOT NULL , PRIMARY KEY ("name", "region"))')
    ('CREATE TABLE "tf2idb_equip_regions" ("id" INTEGER NOT NULL , "region" TEXT NOT NULL , PRIMARY KEY ("id", "region"))')
    ('CREATE TABLE "tf2idb_capabilities"  ("id" INTEGER NOT NULL , "capability" TEXT NOT NULL )')
    Changelog

    10/21/2017 - Updated tf2idb.py to handle blank item_types.

    3/1/2016 - Updated tf2idb.sp and tf2idb.smx - fixed GetItemEquipRegions. Thank you Dagothur.

    12/18/2015 - Updated tf2idb.py to handle the typo in today's schema change. Now all attributes are stored in lowercase.

    11/12/2015 - Updated tf2idb.py for python 3.5 and added item_name column. Thanks to fakuivan.

    10/07/2015 - Updated tf2idb.py to support new schema changes, and not to change the database if parsing failed. Added the table "tf2idb_capabilities" with the columns ("id" INTEGER NOT NULL , "capability" TEXT NOT NULL )
    Attached Files
    File Type: inc tf2idb.inc (2.7 KB, 2930 views)
    File Type: zip sample-db-2015-12-18.zip (410.7 KB, 4677 views)
    File Type: sp Get Plugin or Get Source (tf2idb.sp - 1764 views - 13.4 KB)
    File Type: smx tf2idb.smx (9.3 KB, 5083 views)
    File Type: zip tf2idb.zip (3.5 KB, 2745 views)
    __________________

    Last edited by bottiger; 10-21-2017 at 21:15.
    bottiger is offline
    BatyaMedic
    Member
    Join Date: Sep 2014
    Old 01-22-2015 , 01:59   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #2

    No more crashes and memory leaks like in tf2itemsinfo?Good.
    BatyaMedic is offline
    Chdata
    Veteran Member
    Join Date: Aug 2012
    Location: Computer Chair, Illinois
    Old 01-22-2015 , 04:53   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #3

    PHP Code:
    stock GetSlotFromPlayerWeapon(iClientiWeapon)
    {
        for (new 
    0<= 5i++)
        {
            if (
    iWeapon == GetPlayerWeaponSlot(iClienti))
            {
                return 
    i;
            }
        }
        return -
    1;

    Here's just a small code snippet incase someone needs this one specific thing
    __________________

    Last edited by Chdata; 01-22-2015 at 04:54.
    Chdata is offline
    bottiger
    AlliedModders Donor
    Join Date: Dec 2010
    Old 01-22-2015 , 15:29   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #4

    Quote:
    Originally Posted by Chdata View Post
    PHP Code:
    stock GetSlotFromPlayerWeapon(iClientiWeapon)
    {
        for (new 
    0<= 5i++)
        {
            if (
    iWeapon == GetPlayerWeaponSlot(iClienti))
            {
                return 
    i;
            }
        }
        return -
    1;

    Here's just a small code snippet incase someone needs this one specific thing
    This plugin allows you to query the slot of any item without knowing the player. It is not just limited to weapons and a specific player.
    __________________
    bottiger is offline
    Powerlord
    AlliedModders Donor
    Join Date: Jun 2008
    Location: Seduce Me!
    Old 01-22-2015 , 17:10   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #5

    Glad to see someone finally redid TF2ItemsInfo using SQLite.

    I vaguely remember suggesting this a while ago, but I was too lazy busy to do it myself.

    Edit: On a side note, shouldn't MAX_ATTRIBUTES currently be 15? m_AttributeList has items 000 through 014 according to a netprops dump. As far as I'm aware, CUtlVector netprop tables have the names of their children listed correctly even if their offsets are fake1.

    1You're going to have to trust me on this one, I've done experiments on it. The items in a CUtlVector netprops are just a wrapper around an object2 that stores the position in the CUtlVector that can be used to retrieve its values. I have no idea how this works with items that themselves contain datatables.

    2CSendPropExtra_UtlVector specifically.
    __________________
    Am I back? Well, we'll see.

    Last edited by Powerlord; 01-22-2015 at 17:38.
    Powerlord is offline
    bottiger
    AlliedModders Donor
    Join Date: Dec 2010
    Old 01-22-2015 , 17:34   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #6

    I don't mind changing it to 15, I just copied the number straight from tf2itemsinfo. I also don't know if I did the equip regions correctly.
    __________________
    bottiger is offline
    Powerlord
    AlliedModders Donor
    Join Date: Jun 2008
    Location: Seduce Me!
    Old 01-22-2015 , 18:18   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #7

    A lot of places still have Python2 installed. For instance, most Linux distributions use it.

    On a totally unrelated note, turns out that CUtlVector netprops don't store pointers after all. So... maybe it is possible to get the offsets to the actual values from a CUtlVector netprop... time to take another crack at my cutlvector-netprops branch of SourceMod.
    __________________
    Am I back? Well, we'll see.

    Last edited by Powerlord; 01-22-2015 at 18:19.
    Powerlord is offline
    bottiger
    AlliedModders Donor
    Join Date: Dec 2010
    Old 01-22-2015 , 18:39   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #8

    I did not write the vdf.py library which has Python2 print statements. The code I wrote, tf2idb.py is fully Python3 compatible.

    And yes I realize that calling SQLite for each function call can be slow, but this is the way it should be.

    - I did a huge amount of ground work up to this point already. I do not want to spend any more time than I have to with premature optimizations.
    - To prevent the explosion of handle usage that has doomed tf2itemsinfo to become completely unfixable.
    - Small files are cached in RAM in every modern OS so even if SQLite functions uses disk functions, it will not cause IO.

    I may add a cache for very frequently used natives such as IsValidID. But for everything I will leave it up to someone else here to implement, provided it does not use too many handles.

    Bailopan has already stated he is not interested in continually increasing the handles limit. The handle limit will never be able to go above 32k because it needs to fit in 32bits and 16 of those are used to save the handle type.
    __________________

    Last edited by bottiger; 01-22-2015 at 18:50.
    bottiger is offline
    bottiger
    AlliedModders Donor
    Join Date: Dec 2010
    Old 01-22-2015 , 18:59   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #9

    Quote:
    Originally Posted by Potato Uno View Post
    https://forums.alliedmods.net/showpo...08&postcount=2

    He shows modest interest in obliterating the handle limit. So depending how long that takes, tf2itemsinfo may not stay broken for too long anyway.

    Still, good work, and as for vdf.py, add the __future__ import statement to vdf.py because your script still imports vdf.py (so it does not work on python 3 anyway).
    TF2ItemsInfo is broken for good.

    The handle system is not going anyway any time soon. As Bailopan said, it is a "long bet", and it'll probably be years before a GC is put into sourcepawn.

    Massive handle usage is not the only issue it has. It also has memory leaks, and it is already taking so long to load that it gets killed by the new sourcemod watchdog timer.

    Also if you have already done the work of cleaning up vdf.py, then why don't you post it?
    __________________

    Last edited by bottiger; 01-22-2015 at 19:03.
    bottiger is offline
    Horsedick
    AlliedModders Donor
    Join Date: Sep 2011
    Old 01-22-2015 , 20:19   Re: [TF2] TF2 Item DB (replaces tf2itemsinfo)
    Reply With Quote #10

    How will this interact with other plugins that utilize tf2itemsinfo for a dependency?

    Will the removal of tf2itemsinfo and addition of this be seamless?
    Horsedick is offline
    Reply


    Thread Tools
    Display Modes

    Posting Rules
    You may not post new threads
    You may not post replies
    You may not post attachments
    You may not edit your posts

    BB code is On
    Smilies are On
    [IMG] code is On
    HTML code is Off

    Forum Jump


    All times are GMT -4. The time now is 09:12.


    Powered by vBulletin®
    Copyright ©2000 - 2020, vBulletin Solutions, Inc.
    Theme made by Freecode