Raised This Month: $12 Target: $400
 3% 

[BATCH] FastDL .bz2 recursive bzip2 compress


Post New Thread Reply   
 
Thread Tools Display Modes
Author Message
Chdata
Veteran Member
Join Date: Aug 2012
Location: Computer Chair, Illinois
Old 05-04-2015 , 15:38   [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #1

I got tired of compressing one folder at a time when setting up fastDL .bz2 files and finally got around to just making it compress everything recursively.

For people that don't already have bzip2.exe, I included it.

If you already have it, here's the source for bzip.bat.

Code:
:: Recursive bz2 Compress
:: By: Chdata
:: 4/25/2015

::bzip2.exe -z compress/*.*
@echo off
IF NOT EXIST compress MKDIR compress
for /d /r %%x in (*) do (
    @echo on
    pushd "%%x"
    %~dp0bzip2.exe -z *.*
    popd
    @echo off
)
@echo on
@echo.
@echo Press enter to exit...
@pause >nul
It will try to bzip folders... but for me it always says permission denied when it tries to.
Batch is old and kinda messy so I figured it'd be fine to leave it this way.
Attached Files
File Type: zip bzip.zip (30.6 KB, 2257 views)
__________________

Last edited by Chdata; 05-04-2015 at 15:40.
Chdata is offline
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 05-04-2015 , 17:35   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #2

Is the compress directory even used in your version?

Anyway, here's a Linux version named bzip.sh, although it does require the Bash shell as it uses bash constructs such as GLOBIGNORE.

Apparently you can't attach bz2 files on AlliedModders. Which is a shame as generally Linux distros already have them installed.

Note that this Bash script makes a copy of the files before compressing them. It was either that or do it in place and move the bz2 files after the fact... which would create problems if you had other bz2 files already around.

Code:
#!/bin/bash

GLOBIGNORE="./bzip.sh:./compress"

if [ ! -d compress ]
then
        mkdir compress
fi

cp -r ./* compress

find compress/ -type f ! -name "*.bz2" -exec bzip2 {} +
Attached Files
File Type: zip bzip.zip (284 Bytes, 491 views)
__________________
Not currently working on SourceMod plugin development.
Powerlord is offline
Chdata
Veteran Member
Join Date: Aug 2012
Location: Computer Chair, Illinois
Old 05-04-2015 , 18:38   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #3

The original bzip comes with the compress/ folder in the .rar itself, however my version most likely will search through any folders you have in the same directory as bzip2.exe

I don't really know cause I only have the compress/ folder there, and it works as intended.
__________________

Last edited by Chdata; 05-04-2015 at 18:38.
Chdata is offline
kadet.89
Veteran Member
Join Date: Nov 2012
Location: Serbia
Old 05-07-2015 , 03:32   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #4

does it compress files as well as 7z does? (I mean the compression ratio)

Last edited by kadet.89; 05-07-2015 at 03:33.
kadet.89 is offline
Send a message via Skype™ to kadet.89
Chdata
Veteran Member
Join Date: Aug 2012
Location: Computer Chair, Illinois
Old 05-07-2015 , 08:59   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #5

I tend to only use 7z for map compression since this is easier for multiple files.

7z:
Compression level: Ultra
Method BZip2
Dictionary Size: 900 kb

Both compressions took a pretty long time I guess.

bzip2: 81,944 KB -> 27,478 KB
7z: 81,944 KB -> 26,429 KB

negligible, especially for files that aren't so obtusively large for a download
__________________
Chdata is offline
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 05-07-2015 , 10:16   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #6

If you want Ultra compression with the bzip2 executable, add -9 to its options in the script.
__________________
Not currently working on SourceMod plugin development.
Powerlord is offline
Potato Uno
Veteran Member
Join Date: Jan 2014
Location: Atlanta, Georgia
Old 05-07-2015 , 11:35   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #7

Does bzip2 parallelize the compression for a single file? Or does it use only one thread to compress data?
Potato Uno is offline
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 05-07-2015 , 14:07   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #8

Quote:
Originally Posted by Potato Uno View Post
Does bzip2 parallelize the compression for a single file? Or does it use only one thread to compress data?
The standard bzip2 doesn't, you need the pbzip2 executable for that.

Edit: Some quick googling implies that 7zip also uses the parallel algorithm when creating/extracting bz2 files.

Edit 2: A script for 7zip would have to be rewritten, as the output filename is mandatory in 7zip.
__________________
Not currently working on SourceMod plugin development.

Last edited by Powerlord; 05-07-2015 at 14:18.
Powerlord is offline
Powerlord
AlliedModders Donor
Join Date: Jun 2008
Location: Seduce Me!
Old 05-07-2015 , 14:49   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #9

OK, dunno about for Windows, but here's a Linux/OSX/Cygwin version of bzip.sh that uses 7zip instead.

Assuming 7z is the name of the 7zip executable that is.

Code:
#!/bin/bash

FILE=$(basename $0)

if [ ! -d compress ]
then
        mkdir compress
fi

find . -type f ! -name "*.bz2" ! -name "${FILE}" -print0 | xargs -0 -I % 7z a -tbzip2 compress/%.bz2 %
find . -path "./compress" -prune -o -name "*.bz2" -print0 | xargs -0 -I % cp % compress/%
This probably also works if bzip.sh is in a different directory, as I changed how bzip.sh is removed from the find list.

Note: I didn't redirect its output to /dev/null, so it'll show the status of each file as it comes up.

Edit: Updated to copy any existing bz2 files into the compress folder.
__________________
Not currently working on SourceMod plugin development.

Last edited by Powerlord; 05-07-2015 at 15:13.
Powerlord is offline
Potato Uno
Veteran Member
Join Date: Jan 2014
Location: Atlanta, Georgia
Old 05-07-2015 , 15:23   Re: [BATCH] FastDL .bz2 recursive bzip2 compress
Reply With Quote #10

I would just use a high level VM language like Java or Python that would recursively traverse down a given server folder, locate all files needed (like .bsp), compress them to bz2 with the max compression, and then write out the files to the fastdl directory (or upload them or whatever). It also would allow multiple files to be compressed at once, so there's also the parallelism advantage there.

It would also be cross-platform too. I don't have the time at the moment to throw something together but it should be relatively straightforward, depending how complex you want it to be.
Potato Uno is offline
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 10:25.


Powered by vBulletin®
Copyright ©2000 - 2024, vBulletin Solutions, Inc.
Theme made by Freecode