The new bz2 support for compression of maps

vacs · 05-13-2005, 03:25 AM

I just tried the new Steam build-in HTTP decompression feature and it works nicely.

However as far as I could see it only works with the actual map files (".BSP") and not with other files like the bot navigation files (".NAV").

To compress your BSP files under linux just type:
"bzip2 -z -9 cs_xcom.bsp" (or any other map)

That way bzip2 compresses your file, adds the extension ".bz2" automatically and also deletes the original uncompressed map which is no longer needed.

When someone now connects to your server and he doesn't have the custom map, Steam automatically downloads the bz2 file and decompresses it afterwards.

Thx valve for this as it reduces the download time of extreme large maps considerably, e.g.: "cs_xcom" is reduced from 40mb to 18mb.

05-13-2005, 04:28 AM

Great post.

Fruchtzwerg · 05-13-2005, 05:02 AM

Does it work for HTTP download too? What about other file types like models, sounds and gfx?

vacs · 05-13-2005, 06:24 AM

Quote:

Originally posted by Fruchtzwerg
Does it work for HTTP download too? What about other file types like models, sounds and gfx?

As far as I know it ONLY works for HTTP download and not for the in-build server download streaming.

05-13-2005, 10:52 AM

Awesome, this works beautifully! The maps are considerably smaller (even one 7MB map got compressed to 1.5MB!) and people are commenting about how much faster they are downloading.

This has to be the best extra functionality ever added to Source. Thanks Valve!

BadAss Nox · 05-13-2005, 12:05 PM

Quote:

Originally posted by vacs
As far as I know it ONLY works for HTTP download and not for the in-build server download streaming.

Correct. The srcds actually does compress data sent using the old-style trickle method anyway.

05-13-2005, 01:20 PM

Apparently, the new code also looks for other things in .bz2 format as well.

So-far, I've also also seen sounds attempting to go via bz2.

Knobee

Strike Harder · 05-13-2005, 09:46 PM

Okay, what about windows? What do I need to do.. when the server is empty go and type that for every map.. or do it on my local machine then upload the .bz2 files and delete the old ones or what?

05-14-2005, 09:15 AM

Who do i get this to work for windows ?
how can i bz2 my maps and then upload them to my server ?

05-14-2005, 10:25 AM

Very nice feature, I'm zipping all the custom maps I have on my server!

Fruchtzwerg · 05-14-2005, 01:54 PM

This bz2 support is buggy on client side. After downloading first bzipped extra stuff (models, sounds) it stops downloading.

05-14-2005, 04:05 PM

Ok, update... it seems that you have to bzip EVERYTHING, or it fails to get the non-bzipped files. ie, if you bzip the .bsp and don't bzip the textures, etc, your users will end up with missing textures, as the download will only pull bzip files after it finds the first one.

At least this seems to be the case.

BE CAREFUL!

Knobee

05-14-2005, 06:54 PM

Ok I zipped the map files but for some reason the server doesn't recognize it...
Any idea why?

Strike Harder · 05-14-2005, 08:33 PM

Quote:

Originally posted by simonmichael
Who do i get this to work for windows ?
how can i bz2 my maps and then upload them to my server ?

05-14-2005, 09:16 PM

google "bzip2 windows" and the second hit will give a nice bzip2 tool to download from sourceforge. it's a .exe installer, pretty sweet.

I just ran into that bug, with client only taking one .bz2 file then failing to download the remaining files listed in the .res. I'm ready to freak. We got like 95 maps, i already converted all the .bsp's and deleted the .bsp's from the HTML server cause they seemed to be workin fine. So i moved on to sound files, deleted all the sounds in my clients 'nipperssounds' folder, after converting them and replacing the .wav's on the HTML server. Now each time i join my game server it'll download one .wav.bz2 from the list and join into the game, without finishing downloading the rest of the files listed in the .res. All the files missing from my client exist in .bz2 form on the HTML server, but I can only download one file per time joining the server. So i deleted the blahhh.bsp file on my client, rejoined and it only dl'd the .bsp.bz2 and no sound files, then started dl'ing 1 sound file per joining after that. So we're screwed for anyone joining a map that doesn't have all the files compiled into it's .bsp. They'll get the .bsp but no textures, sounds, models etc.

I'm preying there's a workaround posted here somewhere, otherwise i gotta go through all the original files, and for every map that has more than just a .bsp file, reupload those .bsp files. This is not good.

Knobee, can you verify that you've had multiple .bz2 files download to you when joining your server only once?

05-13-2005, 03:25 AM	#1
vacs Join Date: Dec 2003 Reputation: 0 Posts: 611	The new bz2 support I just tried the new Steam build-in HTTP decompression feature and it works nicely. However as far as I could see it only works with the actual map files (".BSP") and not with other files like the bot navigation files (".NAV"). To compress your BSP files under linux just type: "bzip2 -z -9 cs_xcom.bsp" (or any other map) That way bzip2 compresses your file, adds the extension ".bz2" automatically and also deletes the original uncompressed map which is no longer needed. When someone now connects to your server and he doesn't have the custom map, Steam automatically downloads the bz2 file and decompresses it afterwards. Thx valve for this as it reduces the download time of extreme large maps considerably, e.g.: "cs_xcom" is reduced from 40mb to 18mb. Last edited by vacs: 05-13-2005 at 06:24 AM.

05-13-2005, 09:46 PM	#8
Strike Harder Join Date: Sep 2003 Reputation: 0 Posts: 1,471	Okay, what about windows? What do I need to do.. when the server is empty go and type that for every map.. or do it on my local machine then upload the .bz2 files and delete the old ones or what? Last edited by Strike Harder: 05-14-2005 at 08:33 PM.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

05-13-2005, 04:28 AM	#2
DutchTiger Guest Posts: n/a	Great post.

05-13-2005, 05:02 AM	#3
Fruchtzwerg Banned Join Date: Nov 2003 Reputation: 0 Posts: 310	Does it work for HTTP download too? What about other file types like models, sounds and gfx?

05-13-2005, 10:52 AM	#5
e5.firehazard Guest Posts: n/a	Awesome, this works beautifully! The maps are considerably smaller (even one 7MB map got compressed to 1.5MB!) and people are commenting about how much faster they are downloading. This has to be the best extra functionality ever added to Source. Thanks Valve!

05-13-2005, 01:20 PM	#7
Knobee Guest Posts: n/a	Apparently, the new code also looks for other things in .bz2 format as well. So-far, I've also also seen sounds attempting to go via bz2. Knobee

05-14-2005, 09:15 AM	#9
simonmichael Guest Posts: n/a	Who do i get this to work for windows ? how can i bz2 my maps and then upload them to my server ?

05-14-2005, 10:25 AM	#10
danny9894 Guest Posts: n/a	Very nice feature, I'm zipping all the custom maps I have on my server!

05-14-2005, 01:54 PM	#11
Fruchtzwerg Banned Join Date: Nov 2003 Reputation: 0 Posts: 310	This bz2 support is buggy on client side. After downloading first bzipped extra stuff (models, sounds) it stops downloading.

05-14-2005, 04:05 PM	#12
Knobee Guest Posts: n/a	Ok, update... it seems that you have to bzip EVERYTHING, or it fails to get the non-bzipped files. ie, if you bzip the .bsp and don't bzip the textures, etc, your users will end up with missing textures, as the download will only pull bzip files after it finds the first one. At least this seems to be the case. BE CAREFUL! Knobee

05-14-2005, 06:54 PM	#13
danny9894 Guest Posts: n/a	Ok I zipped the map files but for some reason the server doesn't recognize it... Any idea why?

05-14-2005, 09:16 PM	#15
ArtieCS Guest Posts: n/a	google "bzip2 windows" and the second hit will give a nice bzip2 tool to download from sourceforge. it's a .exe installer, pretty sweet. I just ran into that bug, with client only taking one .bz2 file then failing to download the remaining files listed in the .res. I'm ready to freak. We got like 95 maps, i already converted all the .bsp's and deleted the .bsp's from the HTML server cause they seemed to be workin fine. So i moved on to sound files, deleted all the sounds in my clients 'nipperssounds' folder, after converting them and replacing the .wav's on the HTML server. Now each time i join my game server it'll download one .wav.bz2 from the list and join into the game, without finishing downloading the rest of the files listed in the .res. All the files missing from my client exist in .bz2 form on the HTML server, but I can only download one file per time joining the server. So i deleted the blahhh.bsp file on my client, rejoined and it only dl'd the .bsp.bz2 and no sound files, then started dl'ing 1 sound file per joining after that. So we're screwed for anyone joining a map that doesn't have all the files compiled into it's .bsp. They'll get the .bsp but no textures, sounds, models etc. I'm preying there's a workaround posted here somewhere, otherwise i gotta go through all the original files, and for every map that has more than just a .bsp file, reupload those .bsp files. This is not good. Knobee, can you verify that you've had multiple .bz2 files download to you when joining your server only once?