Raised This Month: $68 Target: $400
 17% 

Source Logging and Reporting Services


Post New Thread Reply   
 
Thread Tools Display Modes
Author
recon0
Veteran Member
Join Date: Sep 2007
Location: US
Plugin ID:
833
Plugin Version:
1.0.2.0
Plugin Category:
Server Management
Plugin Game:
Any
Plugin Dependencies:
    Servers with this Plugin:
     
    Plugin Description:
    Database logging for Source based games (DB version of the standard SRCDS logs) and SourceMod (LogAction, e.g. DB version of the admin logs).
    Old 04-11-2009 , 17:35   Source Logging and Reporting Services
    Reply With Quote #1

    Updates

    Latest SLRD version 1.2.0.0, released 6/6/2009.

    Latest SLRP version 1.0.2.0, released 4/16/2009.


    Note: special characters in DB passwords

    If you use certain special characters in your DB password, they must be escaped in the exe.config file. Here are the characters and their escaped versions:

    quote (") - "
    apostrophe (') - '
    ampersand (&) - &
    less than (<) - &lt;
    greater than (>) - &gt;

    Source for the above escape sequences.



    Background


    Before SLRS, I wrote several other logging systems/readers, and though they worked to a certain degree, they weren't powerful enough. SLRS came about when I noticed some new functions in the API about GameLogHooks. Though I didn't end up using them, the discussion lead to a comment by Tsunami, in which he suggested creating a daemon that can receive a logaddress stream, which is how SLRS works.

    Concept

    SLRS has three major pieces:
    1. SLRD - Source Logging and Reporting Daemon
    2. SLRP - Source Logging and Reporting Plugin
    3. A MySQL Database
    SLRD is responsible for processing log streams from multiple sources. (A server that sends a log stream to SLRD is considered a "source".) The daemon sends everything that you'll see in your standard SRCDS log files (modfolder/logs) to the database.

    SLRP logs all SourceMod LogAction messages to the database.

    With this system in place, your SM admin and server logs are stored in an easily searchable DB table.

    Screenshots

    DB Schema

    Click image for larger version

Name:	diagram.JPG
Views:	1789
Size:	45.4 KB
ID:	40608

    tblLogMessages

    Click image for larger version

Name:	sneakpeaksrls.jpg
Views:	1884
Size:	98.4 KB
ID:	40609


    Requirements
    1. An administrator who knows how to manage and configure MySQL and SourceMod
    2. A MySQL database
    3. A server to run the daemon on
      1. The server must have the Microsoft .NET Framework 2.0 installed
    4. A game server running SourceMod 1.2 or later


    Installing SLRS

    This system is somewhat complex, and requires some technical skills to install it.

    Setting up the database
    1. Create the database "sourceloggingdaemon" with the attached db.sql.txt file
    2. Create a DB account for the the clients to use
      1. This account needs to have access from the machine running SLRD and all game servers running SLRP

    Installing the daemon (SLRD)
    1. Extract the SLRD.zip file to a directory on the server where you want to run the daemon
    2. Set the correct database details in the daemon's SourceLoggingDaemon.exe.config file
    3. Start the SLRD using the following syntax
      1. sourceloggingdaemon.exe -ip listenIPHere -port listenPortHere
    4. If the server you are running SLRD on has a firewall, you will need to add sourceloggingdaemon.exe or listenIP:listenPort (on some firewalls, just the listen port) to the firewall's exception list

    At this point, the logging daemon is online and ready to receive log messages.


    Attaching servers and installing SLRP (repeat these steps for each server you want to attach)
    1. Add the server's IP and Port to the tblSources table. Note the auto-generated SID. You will need it later.
      1. Each server should have it's own row
      2. You have to restart SLRD if you make changes to tblSources so it loads the new version of the table
    2. Add the below line to your server.cfg
      1. log on
    3. Add the daemon's listen IP/Port to the logaddress list for each server that you want to stream logs from by adding the below statement to your server.cfg
      1. logaddress_add listenIPHere:listenPortHere
    4. Download the SLRP.zip package and extract to modfolder/addons
    5. Configure a slrs entry in databases.cfg to point to your SLRS database
    6. Edit sourcemod/configs/slrp/settings.cfg and adjust for your source (set the SID to the SID for this source; you can obtain the SID from the tblSources table)
    7. Load the slrp plugin


    How to use

    All log messages are in tblLogMessages. Use SQL to search the table.


    Changelogs

    SRLD
    • 1.0.0.0
      • Initial Release
    • 1.0.0.1
      • Bug Fix - Fixed null reference exception caused by LogMessage.GetPlayerInfo
    • 1.0.0.2
      • Bug Fix - Safer log parsing
    • 1.0.0.3
      • Bug Fix - Relaxed targeted Regex
      • Bug Fix - Team names can have spaces
    • 1.0.0.4
      • Other - Ensured the proper connection string is used
      • Bug Fix - Adjusted connection string to properly handle UTF8 strings
    • 1.1.0.0
      • Other - UTF insanity resolved
    • 1.1.0.1
      • Bug Fix - Date time parsing region bug (now using ParseExact)
    • 1.2.0.0
      • Feature - Multiple game servers on the same IP address can now stream logs to SLRD
    • Next item
    SLRP
    • 1.0.0.0
      • Initial Release
    • 1.0.1.0
      • Feature - Timestamps are now set correctly for sources that aren't in the DB server's time zone
    • 1.0.2.0
      • Feature - Timestamps are now based on the SourceMod current time
    • Next item


    Troubleshooting

    If SLRD closes shortly after you start it, or if log messages aren't making it into the database, you can check the SRLD logs. The logs are created by default at ./logs.


    Licensing & Source Code

    SLRS is licensed under the GNU GPL 3.0 license. You can download the source code (if you don't know what source code means, you are trying to download the wrong thing) from GitHub.


    Thanks to
    • DJ Tsnuami - For giving me the idea to write this
    • devicenull - For pushing me to parse out more information from the log lines
    • -₪EPIC₪- Trickster |CiC.| - For triggering the MySQL UTF bug
    • XAMPP - For an awesome developer's tool
    Attached Files
    File Type: zip SLRP.zip (8.5 KB, 336 views)
    File Type: sp Get Plugin or Get Source (slrp.sp - 1055 views - 8.4 KB)
    File Type: txt update_utf8.sql.txt (881 Bytes, 320 views)
    File Type: zip SLRD.zip (163.0 KB, 303 views)
    File Type: txt add_messagedt_index.sql.txt (412 Bytes, 336 views)
    File Type: txt db.sql.txt (1.7 KB, 300 views)
    __________________

    Last edited by recon0; 12-26-2013 at 20:49.
    recon0 is offline
    exvel
    SourceMod Donor
    Join Date: Jun 2006
    Location: Russia
    Old 04-13-2009 , 02:03   Re: Source Logging and Reporting Services
    Reply With Quote #2

    This looks fantastic. Great job.
    __________________
    For admins: My plugins

    For developers: Colors library
    exvel is offline
    Send a message via ICQ to exvel
    recon0
    Veteran Member
    Join Date: Sep 2007
    Location: US
    Old 04-13-2009 , 02:40   Re: Source Logging and Reporting Services
    Reply With Quote #3

    Quote:
    Originally Posted by exvel View Post
    This looks fantastic. Great job.
    Thanks.

    I finished coding a new feature in the plugin that sets timestamps correctly for sources that aren't in the DB server's time zone.

    EDIT: SLRP 1.0.1.0 released.
    __________________

    Last edited by recon0; 04-13-2009 at 19:53.
    recon0 is offline
    Hex.
    SourceMod Donor
    Join Date: Feb 2009
    Old 04-14-2009 , 05:32   Re: Source Logging and Reporting Services
    Reply With Quote #4

    Im confused by what this does?

    Sorry
    Hex. is offline
    recon0
    Veteran Member
    Join Date: Sep 2007
    Location: US
    Old 04-14-2009 , 18:38   Re: Source Logging and Reporting Services
    Reply With Quote #5

    Quote:
    Originally Posted by Hex. View Post
    Im confused by what this does?

    Sorry
    I am also confused. Did you read my post?
    Quote:
    Originally Posted by recon0 View Post
    SLRD logs everything that you'll see in your standard SRCDS log files (modfolder/logs) to the database.

    SLRP logs all SourceMod LogAction messages to the database.
    __________________

    Last edited by recon0; 04-14-2009 at 22:11.
    recon0 is offline
    recon0
    Veteran Member
    Join Date: Sep 2007
    Location: US
    Old 04-16-2009 , 20:07   Re: Source Logging and Reporting Services
    Reply With Quote #6

    SLRD 1.1.0.1 and SLRP 1.0.2.0 released!

    See the updates section for details.
    __________________

    Last edited by recon0; 04-16-2009 at 22:29.
    recon0 is offline
    shadenz
    Junior Member
    Join Date: Jul 2006
    Old 05-24-2009 , 21:25   Re: Source Logging and Reporting Services
    Reply With Quote #7

    Any plans for a linux based daemon ?
    shadenz is offline
    recon0
    Veteran Member
    Join Date: Sep 2007
    Location: US
    Old 06-06-2009 , 20:46   Re: Source Logging and Reporting Services
    Reply With Quote #8

    Quote:
    Originally Posted by shadenz View Post
    Any plans for a linux based daemon ?
    You could try using Mono to run SLRD on Linux.


    Update

    SLRD now supports multiple game servers streaming logs from the same IP address.

    __________________

    Last edited by recon0; 09-04-2009 at 16:58.
    recon0 is offline
    Omega_K2
    Junior Member
    Join Date: Aug 2007
    Old 08-06-2009 , 16:39   Re: Source Logging and Reporting Services
    Reply With Quote #9

    Recon, you own
    Omega_K2 is offline
    recon0
    Veteran Member
    Join Date: Sep 2007
    Location: US
    Old 12-15-2009 , 19:55   Re: Source Logging and Reporting Services
    Reply With Quote #10

    Database Schema Update (12/15/09 4:46 PM PST)

    I have added an index for the messageDT column in tbllogmessages. Queries based on that column will return up to 300 times faster (as was the case with a 37 million row table). This is basically a change from O(N) searching to O(log base 2 of N). To apply this update, run add_messagedt_index.sql (attached to the first post). The update will require MySQL to index all the rows in tbllogmessages, which could take quite awhile. If you don't want the speed increase, you do not have to apply this update.
    __________________

    Last edited by recon0; 12-21-2009 at 22:45.
    recon0 is offline
    Reply


    Thread Tools
    Display Modes

    Posting Rules
    You may not post new threads
    You may not post replies
    You may not post attachments
    You may not edit your posts

    BB code is On
    Smilies are On
    [IMG] code is On
    HTML code is Off

    Forum Jump


    All times are GMT -4. The time now is 20:01.


    Powered by vBulletin®
    Copyright ©2000 - 2019, vBulletin Solutions, Inc.
    Theme made by Freecode