Raised This Month: $51 Target: $400
 12% 

MySQL and UTF8 Strings


Post New Thread Reply   
 
Thread Tools Display Modes
claudiuhks
Yam Inside®™℠
Join Date: Jan 2010
Location: Living Randomly
Old 01-28-2014 , 03:25   Re: MySQL and UTF8 strings
Reply With Quote #11

Quote:
Originally Posted by ^SmileY View Post
Change the codification of webpage, maybe this will resolve problems like this.

(I think)
What webpage do you mean? I told you phpMyAdmin which is a webpage created by experts, shows strings inserted by AMX Mod X's MySQL wrong. This should be a MySQL bug, and I hope it will be fixed. Furthermore, I don't understand why MySQL doesn't support wchar_t yet. An Unicode character has the size doubled than an ANSI character. Or, there could be some conversion methods http://utfcpp.sourceforge.net/ that I don't know to use and should be implemented by AMX Mod X development team.

Someone also did report this problem, but I see he didn't say whether he did solve or not StackOverflow.
__________________

Last edited by claudiuhks; 01-28-2014 at 03:33.
claudiuhks is offline
Send a message via MSN to claudiuhks Send a message via Yahoo to claudiuhks Send a message via Skype™ to claudiuhks
^SmileY
Veteran Member
Join Date: Jan 2010
Location: Brazil [<o>]
Old 01-28-2014 , 05:16   Re: MySQL and UTF8 strings
Reply With Quote #12

"codification of webpage" > "codification of your navigator"
__________________
Projects:

- See my Git Hub: https://github.com/SmileYzn
PHP Code:
set_pcvar_num(pCvar, !get_pcvar_num(pCvar)); 
^SmileY is offline
Send a message via MSN to ^SmileY Send a message via Skype™ to ^SmileY
fysiks
Veteran Member
Join Date: Sep 2007
Location: Flatland, USA
Old 01-28-2014 , 05:43   Re: MySQL and UTF8 strings
Reply With Quote #13

Quote:
Originally Posted by ^SmileY View Post
"codification of webpage" > "codification of your navigator"
"encoding" not "codification"
__________________
fysiks is offline
GuskiS
Veteran Member
Join Date: Aug 2007
Location: Latvia
Old 01-28-2014 , 16:15   Re: MySQL and UTF8 strings
Reply With Quote #14

Same mentioned here: http://forums.alliedmods.net/showthread.php?t=94724
__________________
Finished mods:
Trouble in Terrorist Town
MurderMod
The Hidden
Cowboys vs Indians
JailBreak Supreme
Survival Madness
GuskiS is offline
^SmileY
Veteran Member
Join Date: Jan 2010
Location: Brazil [<o>]
Old 01-29-2014 , 08:40   Re: MySQL and UTF8 strings
Reply With Quote #15

Quote:
Originally Posted by GuskiS View Post
not, SET UTF8 NAMES query will force the db to use utf8, not a encoding of navigator to display correct text in phpmyadmin (?)
__________________
Projects:

- See my Git Hub: https://github.com/SmileYzn
PHP Code:
set_pcvar_num(pCvar, !get_pcvar_num(pCvar)); 
^SmileY is offline
Send a message via MSN to ^SmileY Send a message via Skype™ to ^SmileY
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 01-29-2014 , 14:00   Re: MySQL and UTF8 strings
Reply With Quote #16

Quote:
Originally Posted by claudiuhks View Post
What webpage do you mean? I told you phpMyAdmin which is a webpage created by experts, shows strings inserted by AMX Mod X's MySQL wrong.
That's great, except that the text sent between you and phpMyAdmin (and the other direction as well) is determined by the webserver by default. Fun fact: Apache, the most popular webserver in the world, defaults to ISO-8859-1 not UTF-8.

Quote:
Originally Posted by claudiuhks View Post
This should be a MySQL bug, and I hope it will be fixed. Furthermore, I don't understand why MySQL doesn't support wchar_t yet. An Unicode character has the size doubled than an ANSI character. Or, there could be some conversion methods http://utfcpp.sourceforge.net/ that I don't know to use and should be implemented by AMX Mod X development team.

Someone also did report this problem, but I see he didn't say whether he did solve or not StackOverflow.
MySQL supports setting the character type (and collation) per column instead, defaulting to whatever was used when the CREATE DATABASE was issued or whatever the server defaults were when said database was created. (defaults for those are latin1 and latin1_swedish_ci).

This tidbit is relevant in the description of char fields:

Quote:
Originally Posted by MySQL Manual
The following table illustrates the differences between CHAR and VARCHAR by showing the result of storing various string values into CHAR(4) and VARCHAR(4) columns (assuming that the column uses a single-byte character set such as latin1).
(Said table showed 4 for the CHAR columns and the string length + 1 for VARCHAR columns)
__________________
Not currently working on SourceMod plugin development.

Last edited by Powerlord; 01-29-2014 at 14:01.
Powerlord is offline
claudiuhks
Yam Inside®™℠
Join Date: Jan 2010
Location: Living Randomly
Old 01-29-2014 , 21:44   Re: MySQL and UTF8 strings
Reply With Quote #17

Quote:
Originally Posted by Powerlord View Post
(Said table showed 4 for the CHAR columns and the string length + 1 for VARCHAR columns)
My columns are TEXT type and my database and server connection's encoding is set to UTF8MB4, utf8mb4_unicode_ci collation. Do you know how the problem may be fixed?

When inserting characters 畷治 to column (manually with phpMyAdmin), I am able to search matches using Index.PHP?Match=畷治. I think Apache and PHP are alright.
__________________

Last edited by claudiuhks; 01-31-2014 at 22:16.
claudiuhks is offline
Send a message via MSN to claudiuhks Send a message via Yahoo to claudiuhks Send a message via Skype™ to claudiuhks
-Px-
Junior Member
Join Date: Sep 2009
Old 03-01-2014 , 06:04   Re: MySQL and UTF8 strings
Reply With Quote #18

A bit late, but anyway, if someone find this topic via search. There are several caveats in using UTF8 in amxx, they are solvable, but with some drawbacks. By default, amxx use iso8859-1 in communication with mysql, and there is no way to set proper encoding in amxx.
First workaround, if you have full control on DB server, and there is no other clients which need different from UTF8 encoding, you can just add
character-set-server=utf8
collation-server=utf8_general_ci
skip-character-set-client-handshake
to my.cnf and problem should be solved. One case, when its not, will be covered later.
Second way is to specify encoding on connection, and you have two choices: use synchronous api and execute "SET NAMES 'utf8'" right after connection, or use more popular threaded api and prepend needed queries with same SET NAMES 'utf8', but in that case you may be hit by bug in amxx, it doesn't return you InsertId after Insert query and you need to make another query to determine it.
One additional problem, which I discovered recently, is when you have some .inc module called from .sma, and you have strings in non-iso8859-1 encoding (example, declared constant: new const posted_title[] = "Всё вижу...";), then even you save that module in UTF8, compiler generated code(?) reads them as UTF8, but then make some lousy conversion to iso8859-1 so even first amx_log call show them as garbage in log. Didn't investigate this further, just moved them to main .sma code, passed into .inc as variables, and all works fine.
In any case, finding what is wrong without providing the code is hard and time wasting, so its not strange that topic starter didn't solve his issue.
-Px- is offline
Arkshine
AMX Mod X Plugin Approver
Join Date: Oct 2005
Old 04-18-2014 , 16:39   Re: MySQL and UTF8 Strings
Reply With Quote #19

SQL_SetCharset has been added in git3799+. See sqlx.inc for more infos.
__________________
Arkshine is offline
GuskiS
Veteran Member
Join Date: Aug 2007
Location: Latvia
Old 04-18-2014 , 17:13   Re: MySQL and UTF8 Strings
Reply With Quote #20

Finally, thank you
__________________
Finished mods:
Trouble in Terrorist Town
MurderMod
The Hidden
Cowboys vs Indians
JailBreak Supreme
Survival Madness
GuskiS is offline
Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 05:20.


Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Theme made by Freecode