AlliedModders

AlliedModders (https://forums.alliedmods.net/index.php)
-   Snippets and Tutorials (https://forums.alliedmods.net/forumdisplay.php?f=112)
-   -   [BATCH] FastDL .bz2 recursive bzip2 compress (https://forums.alliedmods.net/showthread.php?t=262453)

Chdata 05-04-2015 15:38

[BATCH] FastDL .bz2 recursive bzip2 compress
 
1 Attachment(s)
I got tired of compressing one folder at a time when setting up fastDL .bz2 files and finally got around to just making it compress everything recursively.

For people that don't already have bzip2.exe, I included it.

If you already have it, here's the source for bzip.bat.

Code:

:: Recursive bz2 Compress
:: By: Chdata
:: 4/25/2015

::bzip2.exe -z compress/*.*
@echo off
IF NOT EXIST compress MKDIR compress
for /d /r %%x in (*) do (
    @echo on
    pushd "%%x"
    %~dp0bzip2.exe -z *.*
    popd
    @echo off
)
@echo on
@echo.
@echo Press enter to exit...
@pause >nul

It will try to bzip folders... but for me it always says permission denied when it tries to.
Batch is old and kinda messy so I figured it'd be fine to leave it this way.

Powerlord 05-04-2015 17:35

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
1 Attachment(s)
Is the compress directory even used in your version?

Anyway, here's a Linux version named bzip.sh, although it does require the Bash shell as it uses bash constructs such as GLOBIGNORE.

Apparently you can't attach bz2 files on AlliedModders. Which is a shame as generally Linux distros already have them installed.

Note that this Bash script makes a copy of the files before compressing them. It was either that or do it in place and move the bz2 files after the fact... which would create problems if you had other bz2 files already around.

Code:

#!/bin/bash

GLOBIGNORE="./bzip.sh:./compress"

if [ ! -d compress ]
then
        mkdir compress
fi

cp -r ./* compress

find compress/ -type f ! -name "*.bz2" -exec bzip2 {} +


Chdata 05-04-2015 18:38

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
The original bzip comes with the compress/ folder in the .rar itself, however my version most likely will search through any folders you have in the same directory as bzip2.exe

I don't really know cause I only have the compress/ folder there, and it works as intended.

kadet.89 05-07-2015 03:32

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
does it compress files as well as 7z does? (I mean the compression ratio)

Chdata 05-07-2015 08:59

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
I tend to only use 7z for map compression since this is easier for multiple files.

7z:
Compression level: Ultra
Method BZip2
Dictionary Size: 900 kb

Both compressions took a pretty long time I guess.

bzip2: 81,944 KB -> 27,478 KB
7z: 81,944 KB -> 26,429 KB

negligible, especially for files that aren't so obtusively large for a download

Powerlord 05-07-2015 10:16

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
If you want Ultra compression with the bzip2 executable, add -9 to its options in the script.

Potato Uno 05-07-2015 11:35

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
Does bzip2 parallelize the compression for a single file? Or does it use only one thread to compress data?

Powerlord 05-07-2015 14:07

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
Quote:

Originally Posted by Potato Uno (Post 2294276)
Does bzip2 parallelize the compression for a single file? Or does it use only one thread to compress data?

The standard bzip2 doesn't, you need the pbzip2 executable for that.

Edit: Some quick googling implies that 7zip also uses the parallel algorithm when creating/extracting bz2 files.

Edit 2: A script for 7zip would have to be rewritten, as the output filename is mandatory in 7zip.

Powerlord 05-07-2015 14:49

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
OK, dunno about for Windows, but here's a Linux/OSX/Cygwin version of bzip.sh that uses 7zip instead.

Assuming 7z is the name of the 7zip executable that is.

Code:

#!/bin/bash

FILE=$(basename $0)

if [ ! -d compress ]
then
        mkdir compress
fi

find . -type f ! -name "*.bz2" ! -name "${FILE}" -print0 | xargs -0 -I % 7z a -tbzip2 compress/%.bz2 %
find . -path "./compress" -prune -o -name "*.bz2" -print0 | xargs -0 -I % cp % compress/%

This probably also works if bzip.sh is in a different directory, as I changed how bzip.sh is removed from the find list.

Note: I didn't redirect its output to /dev/null, so it'll show the status of each file as it comes up.

Edit: Updated to copy any existing bz2 files into the compress folder.

Potato Uno 05-07-2015 15:23

Re: [BATCH] FastDL .bz2 recursive bzip2 compress
 
I would just use a high level VM language like Java or Python that would recursively traverse down a given server folder, locate all files needed (like .bsp), compress them to bz2 with the max compression, and then write out the files to the fastdl directory (or upload them or whatever). It also would allow multiple files to be compressed at once, so there's also the parallelism advantage there.

It would also be cross-platform too. I don't have the time at the moment to throw something together but it should be relatively straightforward, depending how complex you want it to be.


All times are GMT -4. The time now is 18:36.

Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.