Raised This Month: $12 Target: $400
 3% 

[TUT] Packed vs Unpacked Strings


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
Bugsy
AMX Mod X Moderator
Join Date: Feb 2005
Location: NJ, USA
Old 04-22-2009 , 01:31   [TUT] Packed vs Unpacked Strings
Reply With Quote #1

Packed vs Unpacked Strings

Recommended pre-reading material if you do not understand how numbers are stored in memory: http://en.wikipedia.org/wiki/Binary_numeral_system

The unpacked string

An unpacked string is the type of string that you commonly (probably always) see used in AMX-X plugins. It is an array of 4-byte cells, each cell representing an individual character in the string. Each character in a string (both packed and unpacked) is represented by an integer value of 0-255. To represent an integer of 0-255, only 1 byte is needed, however, unpacked strings allocate 4-bytes per character leaving 3-bytes of unused memory per cell. Below illustrates this:

Code:
A byte consists of 8-bits, below is a cell of 4-bytes or 32-bits.
This is how a character is stored in an unpacked string cell. 
Only bits 1-8 are used because that is all that is needed to store values 0-255. 

0000 0000 0000 0000 0000 0000 0000 0000 = 0
0000 0000 0000 0000 0000 0000 1111 1111 = 255

The bits in red are un-used\wasted memory. The waste is 3-bytes per cell (24-bits)
The packed string

A packed string is also an array of 4-byte cells but the memory layout is 'packed' so that all memory in the cell is used to store data. This results in the memory requirement for each character to be 1-byte (8-bits) opposed to 4-bytes (32-bits) in an unpacked string. When you declare a packed string you are still declaring an array of 4-byte cells but each cell will hold 4 characters instead of just one as it would in an unpacked string. Note: A packed string can only hold characters from a single-byte character set, such as ascii or one of the extended ascii sets

Code:
This is how a 4-byte cell will appear in packed form. Each color represents 
the storage of an individual character in the string. Notice, nothing is wasted.

This is the packed version of the word "pack"

0111 0000 0110 0001 0110 0011 0110 1011
When should I use a packed string over an unpacked string?

Packed
  • Storing any numerical values in the range of 0 - 255
  • Storing ascii text

Unpacked
  • Storing numerical values > 255
  • Storing unicode text or any non-ascii standard

How do I define and manipulate a packed string in AMX-X?

The usage is almost identical to a normal unpacked string with the exception of accessing the string. You cannot print\manipulate the entire packed string as you can an unpacked. You must first unpack it and then print/format/etc.

Here are two small functions to pack and unpack a string. I don't know how 'expensive' using this unpack function on large strings will be but it's the only option for now. If anyone discovers a way to do this without an additional function please PM me.
PHP Code:
public strpack( const szUnpacked[] , szPacked[] ) 
{     
    new 
i;
    while((
szPacked{i} = szUnpacked[i++])){}
}

public 
strunpack( const szPacked[] , szUnpacked[] ) 
{     
    new 
i;
    while((
szUnpacked[i] = szPacked{i++})){}

To define a string you define it as you normally would with the desired size and add the 'char' identifier after it. You use the same rule of adding 1 to the actual size needed to make room for the null terminating character. You must also include the '!' character before the beginning " when assigning the string to the variable.

PHP Code:
new szString[11 char] = !"The string"
To reference an individual cell in the string, it is the same as with an unpacked array but instead of using [] you use {}.
PHP Code:
Unpacked szString[5];
Packed szString{5}; 
But exactly how much memory does using a packed string save me?

Your memory savings will vary depending on the number and size of the strings in your plugin. I put together a plugin that can quickly determine memory usage based on the number and size of strings in your plugin. By default it is set to use packed strings, to get the result of using unpacked strings, comment the #defined PACKED_STRINGS line.

Here are some results that I came up with using a various number of strings, all with a size of 512. All results are the 'total requirements' value when compiled. I also did a test using size 33 strings and savings can start being noticed at 4+ strings.

Code:
NUM_STRINGS = 1
PACKED = 19756
UNPACKED = 19088
Net savings = -668 bytes [Uses more memory when packed]

NUM_STRINGS = 5
PACKED = 21820
UNPACKED = 27296
Net savings = 5476 bytes

NUM_STRINGS = 10
PACKED = 24400
UNPACKED = 37556
Net savings = 13156 bytes

NUM_STRINGS = 20
PACKED = 29560
UNPACKED = 58076
Net savings = 28516 bytes

NUM_STRINGS = 30
PACKED = 34720
UNPACKED = 78596
Net savings = 43876 bytes

NUM_STRINGS = 50
PACKED   = 45040
UNPACKED = 119636
Net savings = 74596 bytes

NUM_STRINGS = 100
PACKED = 70840
UNPACKED = 222236
Net savings = 151396 bytes
PHP Code:
#include <amxmodx>
#include <amxmisc>

#define PLUGIN "Packed vs Unpacked"
#define VERSION "1.0"
#define AUTHOR "bugsy"

#define PACK_STRINGS

#define NUM_STRINGS    20
#define MAX_STRING_SIZE    512

#if defined PACK_STRINGS
new g_szUnpackBuffer[MAX_STRING_SIZE];
new 
g_szStrings[NUM_STRINGS][MAX_STRING_SIZE char]
#else
new g_szStrings[NUM_STRINGS][MAX_STRING_SIZE]
#endif

public plugin_init() 
{
    
register_plugin(PLUGINVERSIONAUTHOR)
}

public 
plugin_cfg()
{
    for ( new 
iTest iTest NUM_STRINGS iTest++ )
    {
        
#if defined PACK_STRINGS
        
g_szStrings[iTest] = !"Packed strings"    
        
strunpackg_szStrings[iTest] , g_szUnpackBuffer );
        
server_print"%d = %s" iTest g_szUnpackBuffer );
        
#else
        
g_szStrings[iTest] = "Unpacked strings";
        
server_print"%d = %s" iTest g_szStrings[iTest] );
        
#endif
    
}
}

#if defined PACK_STRINGS
public strunpackszPacked[] , szUnpacked[] ) 
{     
    new 
i;

    while((
szUnpacked[i] = szPacked{i++})){}
}
#endif 
Padding

Since a packed string still uses 4-byte cells, you will usually get from 1 to 3 bytes of wasted space. This is because you cannot have only a partial cell. Suppose you want to store 5 letters in a packed string. Theoretically you only need 5 bytes right? since that's what this tutorial has been explaining. Yes, you only need 5 bytes for the storage of your string but you will need 2 cells since 1 cell can only hold 4 characters. In this case we would have 4 bytes used in the first cell for the first 4 letters and then only use 1 byte in the second cell for the remaining letter, plus 1 extra cell for the null character. The left over 2 cells are referred to as 'padding' and will remain there empty holding 0's. The only time this will not occur is when your string size is divisible by 4.

Here are some sample AMX-X scripts using packed strings

PHP Code:
//This will declare a packed and unpacked string, display the values in 
//each cell (which are identical) and then display the size of each string 
//(in cells) at the end.

//The result: Same data 0 = 0 -> Same data 31 = 31
//Packed Size = 9 cells
//Unpacked Size = 33 cells
//Why is the packed 9 cells? See Padding section above.

new szPacked[33 char];
new 
szUnpacked[33];

for ( new 
33 i++ )
{
    
szPacked{i} = i;
    
szUnpacked[i] = i;
    
server_print"Same data %d = %d" szPacked{i} , szUnpacked[i] );
}
    
server_print"Packed Size = %d cells" sizeof szPacked );
server_print"Unpacked Size = %d cells" sizeof szUnpacked ); 
PHP Code:
//This will print out the ascii values for each character
//in the string "Packed"
new szPacked[7 char] = !"Packed";

for ( new 
i++ )
    
server_print"%d" szPacked{i} ); 
        
//This will print the ascii values for "Packed": 80, 97, 99, 107, 101, 100, 0
//P=80 a=97 c=99 k=107 e=101 d=100 NULL=0 
PHP Code:
//Define a packed array used to store bool values for players
new g_szOnline[33 char];

public 
client_putinserver(id)
{
    
g_szOnline{id} = 1;
}

public 
client_disconnect(id)
{
    
g_szOnline{id} = 0;
}

public 
YourFunc(id)
{
    if ( !
g_szOnline{id} )
        
//user not online

__________________

Last edited by Bugsy; 04-30-2009 at 22:48.
Bugsy is offline
danielkza
AMX Mod X Plugin Approver
Join Date: May 2007
Location: São Paulo - Brasil
Old 04-22-2009 , 01:51   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #2

AMXX string functions were specially crafted to avoid the need to convert cells to chars. Doesn't converting cells to chars just to have them convert back to cells seem kinda pointless?
__________________

Community / No support through PM
danielkza is offline
joaquimandrade
Veteran Member
Join Date: Dec 2008
Location: Portugal
Old 04-22-2009 , 01:58   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #3

Quote:
Originally Posted by danielkza View Post
AMXX string functions were specially crafted to avoid the need to convert cells to chars. Doesn't converting cells to chars just to have them convert back to cells seem kinda pointless?
But that's not the only thing you can do with packed strings (to save/use strings) as you can see in the last example.
__________________
joaquimandrade is offline
danielkza
AMX Mod X Plugin Approver
Join Date: May 2007
Location: São Paulo - Brasil
Old 04-22-2009 , 02:00   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #4

Quote:
Originally Posted by joaquimandrade View Post
But that's not the only thing you can do with packed strings (to save/use strings) as you can see in the last example.
Actually conversions will happen anyway. All pawn literals are cells, therefore they need to be truncated to fit in a packed string, and must be re-converted to cells when being used in any statement, AFAIK.

EDIT: Apparently I'm /partially/ wrong, there are special opcodes for packed chars, but still, if you try to use any cell values into expressions conversions have to happen. I'll look into the Pawn manual to see if there are better explanations on the subject.
__________________

Community / No support through PM

Last edited by danielkza; 04-22-2009 at 02:08.
danielkza is offline
Bugsy
AMX Mod X Moderator
Join Date: Feb 2005
Location: NJ, USA
Old 04-22-2009 , 02:01   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #5

Quote:
Originally Posted by danielkza View Post
AMXX string functions were specially crafted to avoid the need to convert cells to chars. Doesn't converting cells to chars just to have them convert back to cells seem kinda pointless?
I know what you're getting at but packed strings can conserve memory if done correctly. You can create a single global unpacked array for a universal unpacking buffer.

It is also beneficial for having an array of bools if you choose not to use bit-fields.

PHP Code:
new g_UnpackBuffer[1024];

public 
Func1()
{
    new 
szPacked1[100 char];

    
strunpackszPacked1 g_UnpackBuffer );
}


public 
Func2()
{
    new 
szPacked2[512 char];

    
strunpackszPacked2 g_UnpackBuffer );

__________________

Last edited by Bugsy; 04-22-2009 at 02:13.
Bugsy is offline
joaquimandrade
Veteran Member
Join Date: Dec 2008
Location: Portugal
Old 04-22-2009 , 02:04   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #6

Quote:
Originally Posted by danielkza View Post
Actually conversions will happen anyway. All pawn literals are cells, therefore they need to be truncated to fit in a pack string, and must be re-converted to cells when being used in any statement, AFAIK.

EDIT: Apparently I'm /partially/ wrong, there are special opcodes for packed chars, but still, if you try to use any cell values into expressions conversions have to happen. I'll look into the Pawn manaul to see if there are better explations on the subject.
Even if you are wrong that was a smart point. I will ask for help to amxx profiler.

Edit:

The results (It's basically the same):
PHP Code:
#include <amxmodx>
#include <fakemeta>

new normal[33]
new 
packed[33 char]

setNormal(x,y)
{
    
normal[x] = y;
}

setPacked(x,y)
{
    
packed{x} = y;
}

getNormal(x)
{
    return 
normal[x];
}

getPacked(x)
{
    return 
packed{x};
}


public 
plugin_cfg()
{
    for(new 
i=0;i<10000;i++)
    {
        new 
random_num(0,32)
        new 
random_num(0,255)
        
        
setPacked(a,b)
        
setNormal(a,b)
        
        
getPacked(a);
        
getNormal(a);
        
        
setPacked(a,0)
        
setNormal(a,0)
        
        
getPacked(a);
        
getNormal(a);
    }
    
    
server_cmd("quit")
    

Code:
date: Tue Apr 21 07:39:32 2009 map: de_dust2
type |                             name |      calls | time / min / max
-------------------------------------------------------------------
   n |                       random_num |      20000 | 0.049776 / 0.000001 / 0.000101
   n |                       server_cmd |          1 | 0.000014 / 0.000014 / 0.000014
   p |                       plugin_cfg |          1 | 0.207914 / 0.207914 / 0.207914
   f |                        setNormal |      20000 | 0.048192 / 0.000001 / 0.000045
   f |                        setPacked |      20000 | 0.048167 / 0.000001 / 0.000092
   f |                        getNormal |      20000 | 0.048578 / 0.000001 / 0.000067
   f |                        getPacked |      20000 | 0.048613 / 0.000001 / 0.000098
0 natives, 0 public callbacks, 1 function calls were not executed.

date: Tue Apr 21 07:40:37 2009 map: de_dust2
type |                             name |      calls | time / min / max
-------------------------------------------------------------------
   n |                       random_num |      20000 | 0.049495 / 0.000001 / 0.000079
   n |                       server_cmd |          1 | 0.000013 / 0.000013 / 0.000013
   p |                       plugin_cfg |          1 | 0.207875 / 0.207875 / 0.207875
   f |                        setNormal |      20000 | 0.048272 / 0.000001 / 0.000079
   f |                        setPacked |      20000 | 0.048243 / 0.000001 / 0.000193
   f |                        getNormal |      20000 | 0.048381 / 0.000001 / 0.000220
   f |                        getPacked |      20000 | 0.048462 / 0.000001 / 0.000080
0 natives, 0 public callbacks, 1 function calls were not executed.
PHP Code:
#include <amxmodx>
#include <fakemeta>

new normal[33]
new 
packed[33 char]

setNormal(x)
{
    
normal[x] = 33 normal[x];
}

setPacked(x)
{
    
packed{x} = 33 packed{x};
}

getNormal(x)
{
    return 
normal[x];
}

getPacked(x)
{
    return 
packed{x};
}


public 
plugin_cfg()
{
    for(new 
i=0;i<10000;i++)
    {    
        new 
random_num(0,32)
        
        
setPacked(a)
        
setNormal(a)
        
        
getPacked(a);
        
getNormal(a);
    }
    
    
server_cmd("quit")
    

Code:
date: Tue Apr 21 07:57:38 2009 map: de_dust2
type |                             name |      calls | time / min / max
-------------------------------------------------------------------
   n |                       random_num |      10000 | 0.026228 / 0.000001 / 0.001046
   n |                       server_cmd |          1 | 0.000011 / 0.000011 / 0.000011
   p |                       plugin_cfg |          1 | 0.104097 / 0.104097 / 0.104097
   f |                        setNormal |      10000 | 0.028161 / 0.000001 / 0.001741
   f |                        setPacked |      10000 | 0.024113 / 0.000001 / 0.000031
   f |                        getNormal |      10000 | 0.024541 / 0.000001 / 0.000118
   f |                        getPacked |      10000 | 0.024453 / 0.000001 / 0.000091
0 natives, 0 public callbacks, 1 function calls were not executed.

date: Tue Apr 21 08:05:27 2009 map: de_dust2
type |                             name |      calls | time / min / max
-------------------------------------------------------------------
   n |                       random_num |      10000 | 0.024472 / 0.000001 / 0.000101
   n |                       server_cmd |          1 | 0.000010 / 0.000010 / 0.000010
   p |                       plugin_cfg |          1 | 0.105678 / 0.105678 / 0.105678
   f |                        setNormal |      10000 | 0.024440 / 0.000001 / 0.000139
   f |                        setPacked |      10000 | 0.023782 / 0.000001 / 0.000051
   f |                        getNormal |      10000 | 0.024079 / 0.000001 / 0.000009
   f |                        getPacked |      10000 | 0.024073 / 0.000001 / 0.000031
0 natives, 0 public callbacks, 1 function calls were not executed.
__________________

Last edited by joaquimandrade; 04-22-2009 at 03:10.
joaquimandrade is offline
Bugsy
AMX Mod X Moderator
Join Date: Feb 2005
Location: NJ, USA
Old 04-22-2009 , 10:05   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #7

Thanks for the profile joaquimandrade +k

It's good to see memory usage can be reduced without an expense to the CPU.

I revised the memory savings section with some results I came up with. Using packed strings is worthwhile depending on the number and size of strings in your plugin.
__________________

Last edited by Bugsy; 04-22-2009 at 11:58.
Bugsy is offline
Exolent[jNr]
Veteran Member
Join Date: Feb 2007
Location: Tennessee
Old 04-23-2009 , 07:38   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #8

I have a situation for an MOTD message that has to be formatted with the plugin.
The array for it is a 2500 array and uses the "len += format(motd[len], ...)" method.
Would using packed strings for this work?
__________________
No private work or selling mods.
Quote:
Originally Posted by xPaw View Post
I love you exolent!
Exolent[jNr] is offline
joaquimandrade
Veteran Member
Join Date: Dec 2008
Location: Portugal
Old 04-23-2009 , 08:38   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #9

Quote:
Originally Posted by Exolent[jNr] View Post
I have a situation for an MOTD message that has to be formatted with the plugin.
The array for it is a 2500 array and uses the "len += format(motd[len], ...)" method.
Would using packed strings for this work?
No because show_motd would still need an unpacked string.
__________________
joaquimandrade is offline
Bugsy
AMX Mod X Moderator
Join Date: Feb 2005
Location: NJ, USA
Old 04-23-2009 , 11:20   Re: [TUT] Packed vs Unpacked Strings
Reply With Quote #10

If you only have one large string that you are trying to conserve space on then the packed approach will not work. Since packed strings can not be manipulated as a whole like unpacked strings can you must first unpack them into a normal unpacked string buffer. In your case this would require you to declare a packed MOTD buffer of 2500 new szMOTD[2500 char] and then a buffer to unpack it new szUnpackBuffer[2500] which is obviously only going to create waste instead of reduce it. See the revised section above titled But exactly how much memory does using a packed string save me?
__________________
Bugsy is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 04:35.


Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Theme made by Freecode