[Solved] Convert UTF-16 LE to UTF-8

Dragokas · **Author**

Hi,

Is there a ready code snippet or something built-in?

I'd like to read localization files, like resource\l4d360ui_english.txt

Thanks.

--
Related topics:
- Bug 5011 - Add native to lookup phrases in valve's translation files
- Getting built-in (valve_*.txt) Translations
- Reverse UTF-8 String! Arabic Chat!

nosoop · 05-13-2021 , 18:20 Re: Convert UTF-16 LE to UTF-8

There's the UCS-2 parsing logic in Doctor McKay's Enhanced Items plugin.

Dragokas · 05-13-2021 , 19:02 Re: Convert UTF-16 LE to UTF-8

Thanks, nosoop.
That works almost good.
Just incorrect handle surrogate pairs. I finished studying algo. Going to fix.

Dragokas · *Last edited by Dragokas; 05-13-2021 at 21:21.*

Ok, after more research / tests, fixed those:

- fixed 2 & 3 bytes character codes handled incorrectly
- added compatibility with surrogate pairs
- fixed missing file's CloseHandle

Not sure, how actual, Doctor McKay seems not active, anyway, created a pull request.

New code is here.

More improvements are welcomed.

The topic is seems solved.

--
Also, made a scheme, as a helper to understand the stuff (in attachment).

References:
https://en.wikipedia.org/wiki/UTF-8
https://en.wikipedia.org/wiki/UTF-16
https://docs.microsoft.com/en-us/dot...g-introduction
https://habr.com/ru/post/544084/
https://unicode.org/faq/utf_bom.html