Most source games use own internal set of translation phrases for ~ 30 languages, located in "resource" folder as plain .txt files (as unpacked / in VPK / or both).
However, you cannot use them directly via SM.
Now, you can!
I'm presenting "Localizer" methodmap designed in a single .inc with well-familiar functions, which you are already know ("overridden" in a way allowing to accept #phrase in arbitrary text position).
Why may I need it?
Just a little real case:
Let's imagine you have L4D plugin with phrase "Easy" (difficulty) and you want to have multiple translations for that phrase.
Instead of asking people to translate it into as many languages as possible, you can find out that a game already has all translations for it in form of "#L4D_DifficultyEasy" phrase, so, you can just use it directly, like this: loc.PrintToChat(client, "#L4D_DifficultyEasy");
This is also especially useful for map names and weapons.
What's inside?
List of methods
Spoiler
List of methods (in methodmap):
- prefix "loc." must be prepended, e.g. loc.PrintToChat()
PHP Code:
void Delegate_InitCompleted(callback func)
void PrintToServer(char[] format, any ...) void ReplyToCommand(int client, const char[] format, any ...) void PrintToChat(int client, const char[] format, any ...) void PrintToChatAll(char[] format, any ...) void PrintHintText(int client, const char[] format, any ...) void PrintHintTextToAll(char[] format, any ...) void PrintCenterText(int client, const char[] format, any ...) void PrintCenterTextAll(char[] format, any ...) void PrintToConsole(int client, const char[] format, any ...) void PrintToConsoleAll(char[] format, any ...) int Format(char[] buffer, int maxlength, const char[] format, any...)
- must be called directly, without prepending methodmap's name (Localizer still should be initialized)
- used to make in-line translation, e.g. PrintToChatAll("%s", Loc_Translate(client, "my #phrase"));
PHP Code:
char[] Loc_Translate(int client, const char[] format, any ...) char[] Loc_TranslateToLang(char[] phrase, int client = LANG_SERVER, char lang_name[] = "", char lang_code[] = "" )
Full description is available in localizer.inc file next to each method. Below documentation may be outdated.
Commands:
PHP Code:
// Lists plugin names that using Localizer API, show API version and installation mode sm_localizer_list
Quick Start
How to add in your plugin?
-------------------------
Minimal sample:
PHP Code:
#include <localizer>
Localizer loc;
public void OnPluginStart() { loc = new Localizer(); // this is always required! }
Localizer methodmap "redefines" well-familiar functions, like PrintToChat(), Format() e.t.c., allowing you to specify #phrase in format buffer argument in an arbitrary place!
Examples:
Spoiler
1) Show translation via console command !t
PHP Code:
#include <localizer>
Localizer loc;
public void OnPluginStart() { loc = new Localizer();
RegConsoleCmd("sm_t", CmdTest); }
public Action CmdTest(int client, int argc) { // You can use it with well-known SM function names, like:
loc.ReplyToCommand(client, "This is auto-translated to client language: #GameUI_Multiplayer");
if( client ) { loc.PrintToChat(client, "%t", "You can surround by special chars as well - Level:#GameUI_Multiplayer!!!"); }
// or even in-line, e.g. for menus (this method is called beyond the methodmap!):
Menu menu = new Menu(); menu.AddItem("0", Loc_Translate(client, "#L4D_DifficultyEasy"), 0); menu.AddItem("1", Loc_Translate(client, "#L4D_DifficultyNormal"), 0); menu.AddItem("2", Loc_Translate(client, "#L4D_DifficultyHard"), 0); menu.AddItem("3", Loc_Translate(client, "#L4D_DifficultyImpossible"), 0); // ...
// or you can retrieve phrase translation directly in a desired language:
// 1. PrintToChatAll(Loc_TranslateToLang("#GameUI_Multiplayer", _, "ukrainian")); // in-line, by full language name
// 2. char buff[192]; loc.TranslateToLang("#GameUI_Multiplayer", buff, sizeof(buff), _, _, "ua"); // via buffer, by language code PrintToChatAll("%s", buff);
return Plugin_Handled; }
Production code:
1) L4D Map Changer (multilingual translation of map and campaign titles)
2) L4D Weapon and melees inc fork (multilingual translation of raziEiL [disawar1] project).
Note: first installation can take up 30 sec. to 5 minutes (refer to "Cautions" section for details). That's happened only once.
To check is Localizer ready, you can use Localizer.IsReady() or via notifier forward (see later).
How to see/find ALL phrases?
- such way, to know, what is available for using...
There are 2 ways:
1) You can dump them with loc.DumpAll() method.
Resulting files are located in "addons/sourcemod/translations/localizer.phrases.txt" (+ each language subfolder),
and they are also compatible with SM's LoadTranslations(), so you have many ways to use it.
2) You can manually see phrases, searching .txt files in "resource" folders. However, it's not a preferred way, because game has multiple "resource" folders, including those bundled in VPK archives.
Naming convention:
- Phrase should always be started from # character.
- It is case sensitive.
- Any character can be prepended to the left side of #phrase (e.g. c#phrase)
- Any* special character can be appended to the end of #phrase (e.g.: #phrase^ #phrase, #phrase$)
- *with one exception: if the special character is one of ".-", the next character should NOT be an alpha-numeric (e.g.: #phrase-a is not correct)
Cautions:
Note 1: don't try to get translation in OnPluginStart(), since it require some time to initialize (10 - 30 sec. at first database re-build, and < 2 sec. next times).
Instead, if you need to access translation as fast as possible, you can register a notification callback function, which is called when localizer is ready to serve you.
Example:
Spoiler
PHP Code:
#include <localizer>
Localizer loc;
public void OnPluginStart() { loc = new Localizer(); loc.Delegate_InitCompleted( OnPhrasesReady ); }
public void OnPhrasesReady() { // in-line translation example PrintToServer("LTranslate = %s", Loc_Translate(LANG_SERVER, "WARNING: #L4D_ServerShuttingDownIdle!!!"));
// example to use with SM LoadTranslations() for %T %t loc.DumpAll(); loc.LoadTranslations();
Also, it is not fully finished (since the war in Ukraine, perhaps, will never be finished).
TF2 is likely not supported due to performance issues in Valve FS. Required integration of VPK_API (raw reader) written by Silvers.
I would be glad to have performance reports from CSGO / TF2 games on slow (non-SSD) servers.
To make such log, run the plugin: performance-test.sp
Then, retrieve a report from from the server console.
Installation methods (for advanced):
Spoiler
Installation is only happened once (until resource updates come out).
Installation method can be selected by passing appropriate optional argument in methodmap initializer, e.g.:
PHP Code:
Localizer loc; loc = new Localizer( LC_INSTALL_MODE_FULLCACHE );
About:
Method #1. LC_INSTALL_MODE_DATABASE
Status: (default)
Features: 2 Levels cache: Database & StringMap.
Benefits: Quite fast query, minimum memory consuming, minimum time for second initialization (after reboot).
Limitations: None.
Method #2. LC_INSTALL_MODE_FULLCACHE
Status: (experimental)
Features: Full cache - StringMap only. All phrases are pre-cached in memory at once.
Benefits: Fastest query.
Limitations: Lot of memory consuming, 5-10 sec. for second initialization.
Method #3. LC_INSTALL_MODE_TRANSLATIONFILE
Status: (experimental, not yet fully implemented for automatic use; you can test it manually, using LC_INSTALL_MODE_FULLCACHE + loc.DumpAll());
Features:
- Dumping phrases to SM compatible .phrases.txt format & LoadTranslation().
- You can use own SM %T %t specifiers, e.g. PrintToChat(client, "%t", "#L4D_ServerShuttingDownIdle");
Benefits: high query speed, minimum time for second initialization.
Limitations:
- perhaps lot of memory consuming.
- Most methods of Localizer methodmap are unavailable.
Method #4. LC_INSTALL_MODE_CUSTOM
Status: (ok)
Features: no installation is performed, no objects are initialized at startup.
Benefits: intended for 2 cases at the moment:
- calling Localizer.Uninstall() method with skipping installation stage.
- calling Localizer.PrecacheTranslationFile() to work with custom translations separately without accessing default intrinsic phrases.
Details on internal implementation (for experts only):
Spoiler
How it works?
Localizer consists of 2 stages:
- Installation (with 3 sub-stages: re-encoding, parsing, caching).
- Using (with 1 or 2 cache subsystems depending on installation method).
1) Installation includes:
- Listing all txt files from "resources" (including contents of VPK).
- Re-encoding UTF-16LE to UTF-8 (results are located in "resources/utf8" subfolder).
- Parsing UTF-8 files into key and value (phrase & translation).
- Moving keys & values to (depending on install method):
in RAM - StringMap (for LC_INSTALL_MODE_FULLCACHE)
dump as SM .phrases.txt format (for LC_INSTALL_MODE_TRANSLATIONFILE)
- Constructing index to store the modification stamp in case resource files are updated to be able incrementally re-build phrases on the next server startup:
* The stamp is represented by a resource file size (due to impossibility for SM to retrieve modification date from VPK container).
There are 3 indices:
Decode index - stored in _index.txt file of resource/utf8 dir
Database index - filenames + stamp are stored in the same database (for LC_INSTALL_MODE_DATABASE)
Translation file index - filenames + stamp are stored in translation .phrases.txt (for LC_INSTALL_MODE_TRANSLATIONFILE) (not yet implemented)
- Encoding & parsing stages are consuming a lot of CPU time (10 - 30 sec.), so they are executed under profiler control with a limitation about 0.3 sec per chunk operation. When this limit exceeded plugin's "thread" is paused for next 0.1 sec and so on.
- When all operations finished, an optional private forward is called to notify the basic code of the translator's readiness.
- To prevent multiple plugins from performing encode/cache operations simultaneously at server startup, the inter-plugin synchronization mechanism is implemented via ConVar:
First loaded plugin is registering ConVar, changing it to a signal state as soon as it finishes operations.
Other plugins check for ConVar, if it exists they're hooking for change event, waiting for a signal state, and then continue its operations, returning ConVar back to the WAIT state, so next plugins in the chain will wait for its init completion.
In parallel, the watchdog timer is created to prevent incidental deadlock.
2) Using (described for LC_INSTALL_MODE_DATABASE):
- When you query for a phrase translation, Localizer firstly look in cache L2 - which is StringMap (by default, it is empty).
- If not found, it makes a non-threaded (sync) call to cache L1 - which is a local SQLite database.
- Results for a single phrase are retrieved then for ALL languages and they are get cached in L2, making all further calls to same phrase to be as fast as possible, doesn't matter the client from which country queried for it the next time.
- If a phrase doesn't found in a selected language, one more attempt is made to retrieve it using a default server language. If both failed, the original alias stays untouched, unless otherwise specified in documentation.
- L2 cache (RAM) is destroyed on plugin shutdown or calling Localizer.Close() method manually.
Compatibility:
L4D1
L4D2
CS:GO
TF2 (not supported due to performance issues, however, you may try)
Requirements:
- SM 1.10+
- SQLite Extension (included in SourceMod)
- Make sure, you have a correct file: addons/sourcemod/cfg/languages.cfg (from one of SM 1.10+ packages)
- Make sure, you have correct default database settings in: addons/sourcemod/cfg/databases.cfg (which is coming with SourceMod)
How to uninstall
- to uninstall completely, please, call loc.Uninstall() method
PHP Code:
#include <localizer>
public void OnPluginStart() { Localizer loc = new Localizer(LC_INSTALL_MODE_CUSTOM); loc.Uninstall(); }
Or use can just delete folder {game}/resource/utf8, however this one will not erase database records.
- Externet - for the most amazing IDE BasicPawn and its excellent support, which has greatly simplified the development of this include.
- Wend4r and KyleSanderson - for suggesting walkaround against IDatabase leak while too many single threaded queries are queued - #1505.
- asherkin - for explaining multiple DB result sets (no, they aren't used here), influencing on correct Lock/Unlock usage.
- R1KO & CrazyHackGUT - for the manual and comments on various database stuff.
- SourceMod dev. team - for nice documentation, partially used here.
FAQ:
Q: Why the first run takes a much time?
A: At the first run, the installation process is performed. Next run will take < 1 sec.
Q: Ok, but why so long on HDD Drive?
A: 95% of the time had waste solely on resources disk read operation and database disk write operation. I can't do anything to improve it.
localizer.inc(498) : warning 203: symbol is never used: "file"
// 498 | public bool PrecacheTranslationFile(char[] file, LC_CACHE cache = LC_CACHE_RAM)
localizer.inc(498) : warning 203: symbol is never used: "cache"
// 498 | public bool PrecacheTranslationFile(char[] file, LC_CACHE cache = LC_CACHE_RAM)
localizer.inc(1075) : warning 204: symbol is assigned a value that is never used: "total_time"
localizer.inc(1136) : warning 204: symbol is assigned a value that is never used: "total_time"
0.94
- Profiler is auto-enabled when the first installation occurs
- Display on server and chat the progress of installation per each file being parsed
- Fixed warnings in sm 1.12
e.g.
Code:
PrintToServer(">>> [Localizer] Beginning transformation of language files...");
PrintToServer(">>> [Localizer] It may freeze the server. Don't panic!");
PrintToServer(">>> [Localizer] This is one-time procedure.");
and messages per each file while parsing > on server and chat, incl. when new addon map is installed.
Linear Mode
