[recon0]

Updates

Latest SLRD version 1.2.0.0, released 6/6/2009.

Latest SLRP version 1.0.2.0, released 4/16/2009.


Update (12/15/09 6:01 PM PST)

Updated db.sql.txt to include the new updates.


Database Schema Update (12/15/09 4:46 PM PST)

I have added an index for the messageDT column in tbllogmessages. Queries based on that column will return up to 300 times faster (as was the case with a 37 million row table). This is basically a change from O(N) searching to O(log base 2 of N). To apply this update, run add_messagedt_index.sql (attached). The update will require MySQL to index all the rows in tbllogmessages, which could take quite awhile. If you don't want the speed increase, you do not have to apply this update.


SLRD Updated (6/6/2009 5:45 PM PST)

SLRD now supports multiple game servers streaming logs from the same IP address.


UTF insanity - an answer (finally) (4/18/09 - 2:05 PM PST)

Thanks to Tonci Grgin at MySQL, I finally know what the UTF problem was. The update script was wrong. After running some diagnostics suggested by him, I realized that MySQL WB didn't create the update script was expecting, and the update script I was expecting wouldn't have worked either.

THERE IS NOT A BUG IN MySQL 5.1.x

BACKUP YOUR DB and then run update_utf8.sql if your initial creation of your DB was on or before 4/12/09, or if you are getting strange MySQL exceptions in SLRD.


SLRP updated (4/16/09 - 7:09 PM PST)

SLRP now uses the SourceMod current time (sm_time_adjustment will effect the timestamps) when inserting SM events. The gmtOffset config value is no longer used.


Date time parsing bug fixed (4/16/09 - 4:58 PM PST)

I have fixed a date time parsing bug that effects SLRD servers that aren't in the US (the error was found with a UK based server). The reason it effects non US based servers is because of region differences with the .NET Framework (the .NET framework did exactly what it's supposed to do).


Special characters in DB passwords (4/13/09 - 2:49 PM PST)

If you use certain special characters in your DB password, they must be escaped in the exe.config file. Here are the characters and their escaped versions:

quote (") - "
apostrophe (') - '
ampersand (&) - &
less than (<) - &lt;
greater than (>) - &gt;

Source for the above escape sequences.


IMPORTANT UPDATE (4/12/09 - 3:55 PM PST)

After some very annoying UTF issues caused by MySQL version differences between my production and testing machines, I have rolled back some changes in older revisions. Everyone should run update_utf8.sql and install SLRD 1.1.0.0 immediately. There is a bug (EDIT: there isn't a bug) with MySQL server 5.1.x where UTF strings can't be inserted by the Net Connector (bug report filed with MySQL AB). Until the bug is fixed, please use MySQL Server 5.0.x.


Background

I've written three logging systems (two attempts with SourceLogs) and the released log file reader SourceLogFileReader.

None of those systems were ever used on live servers, and only one was ever tested with a live server.

I noticed some new functions in the Scripting API relating to GameLogHooks. That got me thinking about creating a decent logging system for SRCDS based servers.

After investigating the functions, I found that they didn't do what I wanted. Then, Tsunami suggested creating a daemon that can recieve a logaddress stream. After he said it, I thought, prefect, that will work.

A few days of coding later, I present to you, SLRS: Source Logging and Reporting Services.


Concept

SLRS is a multi-client, multi-server solution.

SLRS has three major pieces:

1. SLRD - Source Logging and Reporting Daemon
2. SLRP - Source Logging and Reporting Plugin
3. A MySQL Database

SLRD is responsible for processing log streams from multiple sources. A server that sends a log stream to SLRD is considered a "source".

SLRD logs everything that you'll see in your standard SRCDS log files (modfolder/logs) to the database.

SLRP logs all SourceMod LogAction messages to the database.

With this system in place, your SM admin and server logs are stored in an easily searchable DB table.

Screenshots

DB Schema



tblLogMessages




Requirements

1. An administrator who knows how to manage and configure MySQL and SourceMod
2. A MySQL database
3. A server to run the daemon on
   1. The server must have the Microsoft .NET Framework 2.0 installed
4. A game server running SourceMod 1.2 or later

Installing SLRS

This system is somewhat complex, and requires some technical skills to install it.

Setting up the database

1. Create the database "sourceloggingdaemon" with the attached db.sql.txt file
2. Create a DB account for the the clients to use
   1. This account needs to have access from the machine running SLRD and all game servers running SLRP

Installing the daemon (SLRD)

1. Extract the SLRD.zip file to a directory on the server where you want to run the daemon
2. Set the correct database details in the daemon's SourceLoggingDaemon.exe.config file
3. Start the SLRD using the following syntax
   1. sourceloggingdaemon.exe -ip listenIPHere -port listenPortHere
4. If the server you are running SLRD on has a firewall, you will need to add sourceloggingdaemon.exe or listenIP:listenPort (on some firewalls, just the listen port) to the firewall's exception list

At this point, the logging daemon is online and ready to receive log messages.


Attaching servers and installing SLRP (repeat these steps for each server you want to attach)

1. Add the server's IP and Port to the tblSources table. Note the auto-generated SID. You will need it later.
   1. Each server should have it's own row
   2. You have to restart SLRD if you make changes to tblSources so it loads the new version of the table
2. Add the below line to your server.cfg
   1. log on
3. Add the daemon's listen IP/Port to the logaddress list for each server that you want to stream logs from by adding the below statement to your server.cfg
   1. logaddress_add listenIPHere:listenPortHere
4. Download the SLRP.zip package and extract to modfolder/addons
5. Configure a slrs entry in databases.cfg to point to your SLRS database
6. Edit sourcemod/configs/slrp/settings.cfg and adjust for your source (set the SID to the SID for this source; you can obtain the SID from the tblSources table)
7. Load the slrp plugin

How to use

All log messages are in tblLogMessages. Use SQL to search the table.


Changelogs

SRLD

• 1.0.0.0
  • Initial Release
• 1.0.0.1
  • Bug Fix - Fixed null reference exception caused by LogMessage.GetPlayerInfo
• 1.0.0.2
  • Bug Fix - Safer log parsing
• 1.0.0.3
  • Bug Fix - Relaxed targeted Regex
  • Bug Fix - Team names can have spaces
• 1.0.0.4
  • Other - Ensured the proper connection string is used
  • Bug Fix - Adjusted connection string to properly handle UTF8 strings
• 1.1.0.0
  • Other - UTF insanity resolved
• 1.1.0.1
  • Bug Fix - Date time parsing region bug (now using ParseExact)
• 1.2.0.0
  • Feature - Multiple game servers on the same IP address can now stream logs to SLRD
• Next item

SLRP

• 1.0.0.0
  • Initial Release
• 1.0.1.0
  • Feature - Timestamps are now set correctly for sources that aren't in the DB server's time zone
• 1.0.2.0
  • Feature - Timestamps are now based on the SourceMod current time
• Next item

Troubleshooting

If SLRD closes shortly after you start it, or if log messages aren't making it into the database, you can check the SRLD logs. The logs are created by default at ./logs.


Licensing & Source Code

SLRS is licensed under the GNU GPL 3.0 license. You can download the source code (if you don't know what source code means, you are trying to download the wrong thing) from SVN. The SVN URL is https://svn.microbits.info:444/slrs/trunk. The public ro username and password is anonsvn.


Thanks to

• DJ Tsnuami - For giving me the idea to write this
• devicenull - For pushing me to parse out more information from the log lines
• -₪EPIC₪- Trickster |CiC.| - For triggering the MySQL UTF bug
• XAMPP - For an awesome developer's tool

[exvel]

This looks fantastic. Great job.

[recon0]

Originally Posted by exvel This looks fantastic. Great job.

Thanks.

I finished coding a new feature in the plugin that sets timestamps correctly for sources that aren't in the DB server's time zone.

EDIT: SLRP 1.0.1.0 released.

[Hex.]

Im confused by what this does?

Sorry

[recon0]

Originally Posted by Hex. Im confused by what this does? Sorry

I am also confused. Did you read my post?

Originally Posted by recon0 SLRD logs everything that you'll see in your standard SRCDS log files (modfolder/logs) to the database. SLRP logs all SourceMod LogAction messages to the database.

[recon0]

SLRD 1.1.0.1 and SLRP 1.0.2.0 released!

See the updates section for details.

[shadenz]

Any plans for a linux based daemon ?

[recon0]

Originally Posted by shadenz Any plans for a linux based daemon ?

You could try using Mono to run SLRD on Linux.


Update

SLRD now supports multiple game servers streaming logs from the same IP address.

[Omega_K2]

Recon, you own

[recon0]

Database Schema Update (12/15/09 4:46 PM PST)

I have added an index for the messageDT column in tbllogmessages. Queries based on that column will return up to 300 times faster (as was the case with a 37 million row table). This is basically a change from O(N) searching to O(log base 2 of N). To apply this update, run add_messagedt_index.sql (attached to the first post). The update will require MySQL to index all the rows in tbllogmessages, which could take quite awhile. If you don't want the speed increase, you do not have to apply this update.