Map Optimization Program
Cagey
Ex-Unknown Worlds Programmer Join Date: 2002-11-15 Member: 8829Members, Retired Developer, NS1 Playtester, Constellation
Ex-Unknown Worlds Programmer Join Date: 2002-11-15 Member: 8829Members, Retired Developer, NS1 Playtester, Constellation
<div class="IPBDescription">helps fix MAX_MAP_PLANES error!</div> I'm hitting pretty close to the map planes limit (32767) on the map I'm working on, and it seemed silly that a map with 8,000 faces would require 28,000 planes (some are also used just by nodes and clipnodes, but it still seemed excessive). It turns out that the array of planes isn't garbage collected when csg fills the exterior portion of a non-leaking level... as a result, there are thousands of unused planes that are stored in the final BSP that aren't actually used for anything. I've written a small program that checks for and removes unused planes from maps, reducing the count and potentially taking a map that was over the limit back to the safe range (instructions on how to do that later in this post).
Since a lot of unnecessary data is being dropped, the map files shrink in size pretty dramatically, too. Some test results:
map name #planes before / after size (KB) before / after
bast 32746 / 11852 5077 / 4669
caged 18554 / 7507 5626 / 5410
eclipse 18820 / 6468 5929 / 5688
hera 32656 / 10807 5841 / 5415
tanith 30476 / 11379 5177 / 4804
nancy 27108 / 9762 5518 / 5179
nothing 22166 / 8079 6623 / 5948
Your milage may vary, but as you can see, it has a dramatic effect on the total planes in the map, usually reducing the count by over 60%.
<span style='color:green'>EDIT: I've replaced the version that was posted here with one that's slightly more efficient, and I've also posted modified compile tools that allow you to go over the 32767 planes limit when you compile your bsp. The modified tools have no special requirements for use -- they just moved MAX_MAP_PLANES to 65535 internally. You can download everything from <a href='http://www.unknownworlds.com/forums/index.php?act=ST&f=4&t=21248' target='_blank'>here</a>.</span>
To use the program, you can drag and drop your bsp file over it. The program overwrites the original file, so make a backup if you want to preserve the unoptimized copy. **Note -- since this program shrinks the map, it will checksum differently -- if a server is running the unoptimized copy and you have the optimized, it might not let you connect **.
I'm going to talk to Merl about rolling this optimization into the tools themselves. Don't worry when the program takes under a second to execute... it's really that fast!
Enjoy, and up the complexity of those maps!
Since a lot of unnecessary data is being dropped, the map files shrink in size pretty dramatically, too. Some test results:
map name #planes before / after size (KB) before / after
bast 32746 / 11852 5077 / 4669
caged 18554 / 7507 5626 / 5410
eclipse 18820 / 6468 5929 / 5688
hera 32656 / 10807 5841 / 5415
tanith 30476 / 11379 5177 / 4804
nancy 27108 / 9762 5518 / 5179
nothing 22166 / 8079 6623 / 5948
Your milage may vary, but as you can see, it has a dramatic effect on the total planes in the map, usually reducing the count by over 60%.
<span style='color:green'>EDIT: I've replaced the version that was posted here with one that's slightly more efficient, and I've also posted modified compile tools that allow you to go over the 32767 planes limit when you compile your bsp. The modified tools have no special requirements for use -- they just moved MAX_MAP_PLANES to 65535 internally. You can download everything from <a href='http://www.unknownworlds.com/forums/index.php?act=ST&f=4&t=21248' target='_blank'>here</a>.</span>
To use the program, you can drag and drop your bsp file over it. The program overwrites the original file, so make a backup if you want to preserve the unoptimized copy. **Note -- since this program shrinks the map, it will checksum differently -- if a server is running the unoptimized copy and you have the optimized, it might not let you connect **.
I'm going to talk to Merl about rolling this optimization into the tools themselves. Don't worry when the program takes under a second to execute... it's really that fast!
Enjoy, and up the complexity of those maps!
Comments
For QuArK users, here's how to edit your commandline to get that -maxplanes 65535 in:
1. With your map open, goto Half-life -> Customize Menu
2. Choose the compile type you want to add this to from the list on the left (Go and Complete Rebuild are your best choices)
3. On the right, in the additional arguments dropbox field next to each part of the compile process, type in "-maxplanes 65535" for each part of the process
4. It will only work for the compile types that you added it to, so you may wanna go nuts and add it to all
Make sure you've got the right version of the tools... Merl's customization v1.7 of Zoner's 2.5.3
<a href='http://collective.valve-erc.com/index.php?go=mhlt&PHPSESSID=462f1fb79536e778630f9880bb0598e0' target='_blank'>Merl's custom build v1.7</a>
<a href='http://moore.lorikeet.id.au/vdc/mhlt/instructions.html' target='_blank'>Instruction Manual including -maxplanes</a>
<!--QuoteBegin--></span><table border='0' align='center' width='95%' cellpadding='3' cellspacing='1'><tr><td><b>QUOTE</b> </td></tr><tr><td id='QUOTE'><!--QuoteEBegin-->Variable limits are a higly experimental advanced feature, and should only really be used by people who know what they are doing. The limits that are set in the tools are set for good reasons, not just so that they can annoy mappers; if you wish to change these limits, you really must understand the consequences of doing so. While the compile tools do not modify any part of your .map file, and hence cannot harm your map in any way, playing with these limits willy nilly may waste compiling time.
This edition of the tools comes with two commandline based variable limits: -maxplanes N and -maxclipnodes N, which change the MAX_MAP_PLANES and MAX_MAP_CLIPNODES limits respectively (where N is the new limit).
The default MAX_MAP_PLANES limit is 32767. Increasing this limit may cause the engine to drop faces from brush based entities, and cause texture anomalies in software rendering mode. Decreasing this limit has no apparent side effect.
The default MAX_MAP_CLIPNODES limit is 32767. This limit cannot be increased past 2147483646 due to hard limitations within this version of the compile tools. The increase of this limit is largley undocumented. Decreasing the limit has no apparent side effect.<!--QuoteEnd--></td></tr></table><span class='postcolor'><!--QuoteEEnd-->
After bending the rules by upping max planes in Merl's, my program can drop you back down again.
** Command: Change Directory
** Parameters: "J:\HL"
** Executing...
** Command: C:\PROGRA~1\VALVEH~1\TOOLS\MERL1~1.7\HLCSG.EXE
** Parameters: "h:\mapedit\- maps\selfmade\ns_ollj9" -maxplanes 65535 -low -hullfile j:\hl\ns\nshulls.txt
Unknown option "-maxplanes"
hlcsg v2.5.3 rel Custom Build 1.7 (Dec 9 2002)
Zoner's Half-Life Compilation Tools -- Custom Build
Based on code modifications by Sean 'Zoner' Cavanaugh
Based on Valve's version, modified with permission.
Submit detailed bug reports to (merlinis@bigpond.net.au)
-= hlcsg Options =-
-nowadtextures : include all used textures into bsp
-wadinclude file : place textures used from wad specified into bsp
-noclip : don't create clipping hull
-noclipeconomy : turn clipnode economy mode off
-onlyents : do an entity update from .map to .bsp
-noskyclip : disable automatic clipping of SKY brushes
-tiny # : minmum brush face surface area before it is discarded
-brushunion # : threshold to warn about overlapping brushes
-hullfile file : Reads in custom collision hull dimensions
-texdata # : Alter maximum texture memory limit (in kb)
-chart : display bsp statitics
-low | -high : run program an altered priority level
-nolog : don't generate the compile logfiles
-threads # : manually specify the number of threads to run
-estimate : display estimated time during compile
-verbose : compile with verbose messages
-noinfo : Do not show tool configuration information
-nonulltex : Turns off null texture stripping
-dev # : compile with developer message
-wadconfig name : Specify a configuration to use from wad.cfg
-wadcfgfile path : Manually specify a path to the wad.cfg file
-wadautodetect : Force auto-detection of wadfiles
mapfile : The mapfile to compile
** Executing...
** Command: C:\PROGRA~1\VALVEH~1\TOOLS\MERL1~1.7\HLBSP.EXE
** Parameters: "h:\mapedit\- maps\selfmade\ns_ollj9" -maxplanes 65535 -low -maxnodesize 2048 -leakonly
Unknown option "-maxplanes"
hlbsp v2.5.3 rel Custom Build 1.7 (Dec 9 2002)
Zoner's Half-Life Compilation Tools -- Custom Build
Based on code modifications by Sean 'Zoner' Cavanaugh
Based on Valve's version, modified with permission.
Submit detailed bug reports to (merlinis@bigpond.net.au)
-= hlbsp Options =-
In the meantime, I guess you'll be stuck with the error at 32767 a little longer.
I'm trying to help people, the program that I wrote does exactly what it's supposed to--lowers the number of planes and the size of maps... The problem with the compile tools came when I trusted another program's manual, which was apparently not accurate (it's quoted above). I even apologized to you for my mistake and edited the top post so people wouldn't be misled.
Unless I'm not understanding a joke, I really don't see why you need to make this personal.
anyway, i still dont know WHAT these planes are <!--emo&:(--><img src='http://www.unknownworlds.com/forums/html/emoticons/sad.gif' border='0' valign='absmiddle' alt='sad.gif'><!--endemo-->
(im a noob, remember?)
so could anyone explain me what i can do to avoid this?
is it something with textures?
entitys?
or just walls?
The bsp files normally contain the plane data for the "outside" of the map--even though the outer brush surfaces have been removed by qbsp. Since they aren't used and are basically taking up space, my program removes these "outside" planes (actually, what it does is count the number of times every plane is used by the map, and then remove the ones that have zero references--the "outside" case is the most common cause for an unused plane).
Don't think so. I believe the compiler creates a plane for each brush, even if another brush already uses the same plane.
And XP-Cagey: Excellent tool, once it works. I'll have to do some experimentation with it, but it seems like it may help us QuArK users a lot by turning the mitered wall-maker duplicator into a useful tool.
Explenation:
The wall-maker duplicator (with "mitered corners" checked) will treat all brushes placed inside it like the "inside" of the level and create walls with mitered corners around it to form the hull of that level. Pretty much like the Unreal editor works.
The downside of this is that all those mitered corners create an enormous amount of unused planes. Now your tool seems to take that liability away. Quite nice indeed <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' valign='absmiddle' alt='smile.gif'><!--endemo-->
A simple cube has 6 faces, so it has 6 planes in the first "hull", which controls how the map looks.
It also has 14 more planes in the three clipping hulls (6 for onos hull + 6 for marine/fade + 2 more for different the heights of marine crouch/lerk/skulk/gorge, which shares its "walls" with marine standing since they have the same width and length), but that's something that I'd rather not get into at the moment <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' valign='absmiddle' alt='smile.gif'><!--endemo-->.
If you line up two sets of brushes that are the same size, they will share planes. I'm attaching three sample maps for you to look at -- the first one is the cube example, and totals 20 faces after it's optimized. The second one puts a second cube on the map, not lined up with the first. The number of planes doubles. The third map lines up the cubes--if you can draw a straight line along two faces, they share a plane. The unoptimized counts are 100 for the single cube, 200 for both, and 130 for both when they're lined up, so you save 70 unoptimized planes by having the second cube in a line with the first. You can play around with the sample maps to see what happens when you move a single wall or move an entire cube in a single direction. Run the compiled maps on my program and compare the numbers.
The optimized counts for the three maps attached are 20, 43, and 28, so the savings is almost as good (only 8 new planes instead of 23 when they're lined up) after optimization. For the record, I'm not sure why the optimized count was 43 instead of 40 in the second case--I just mark the fact that 43 were used with the program, I don't test why they are used--I actually expected to see 40. If someone wants to theorize on this, I'd love to hear an explanation <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' valign='absmiddle' alt='smile.gif'><!--endemo-->.
He's added an awful lot to the tools, and others have contributed lots of fixes as well into the package, but nothing to do with increasing planecounts or getting around the limit like this. Not to mention the topic of getting around the limit is usually shot down and dismissed as impossible, so it's never been discussed at length.
I'm inclined to think XP-Cagey is pulling your legs, end of story.
I talked with Merl this morning via email. You might want to do the same if you don't believe me. His email is listed on collective.valve-erc.com.
The manual link above was taken from an article on collective.valve-erc.com announcing custom build 1.7. Merl said that the manual was actually a manual listing features for build 1.7.1 which isn't publicly available. He took down the public access to it to prevent further confusion. He said there is no ETA on the release of 1.7.1, and that it might not be the next public release number. He also invited me to give him my source (which I did) so he can integrate my changes into future versions of the tools once they pass beta testing.
I have nothing to gain by "pulling" anything, and I'm saddened that you're so quick to jump to that conclusion, especially when you could simply <i>download and try</i> what I've written to see that it works. You can use Valve's own bspinfo program from their 2.3 SDK downloaded from any site on the net to confirm the before and after numbers I'm claiming... unless you think Valve is pulling your leg too. Or you can just look at the file size difference from before/after and realize that I'm definately cutting some information out of the map before you play it and see that it still works 100% in the game.
I'll be using this on my map, and I wanted to help other mappers since this seems to be a limit that other people are hitting as well. I never claimed that I could extend the total planes that the half-life engine can use; I claimed that I could reduce the plane count in maps at a given complexity, so that maps that the compile tools think would break are still workable.
I haven't gotten Merl's permission to post our email conversation, or I'd do so... I'm extremely irritated about the reaction some people are having to my attempted generosity since I hold trustworthiness in high regard.
Cash -- there aren't any command line options, since it only does the one thing...
Useage: opt_plns <full bsp filename including path>
On Win2000 systems (and I think other versions of Windows), you should be able to simply drag the compiled bsp over the program icon and let it go to start the program. You should get a little report that looks like bspinfo followed by a line saying how many planes were actually used and a line that says "Optimized!".
EDIT: FYI, if there are any spaces in the path to your file, you'll need to put the path inside double quotes (which is standard for command line progams under Win2000).
Follow <a href='http://collective.valve-erc.com/index.php?go=mhlt' target='_blank'>this link</a>, click the readme.txt, notice that -maxplanes is in the 1.7.1 changes with a warning that it's experimental. It even asks beta users to refer to what I posted above.
If Merl doesn't add -maxplanes, I always can if I get his permission to merge it--the static array used to hold plane information in the programs needs to be replaced with malloc'd storage so that it can change sizes at runtime, and that space needs to be free'd when the programs finish. It's not a difficult task.
To see if my claim about reducing the planes is valid, half compile your map using HLCSG and HLBSP only. Use HLVIS -fast -chart to get the number of planes output from the compilers. Run the bsp file through my program. Use HLVIS -fast -chart again and see that it reports the same change that my program does.
Once the map is optimized, HLCSG -onlyents and HLRAD will break (make the optimization the final step when you're compiling), but it will run fine in the game.
Cagey, your tool rocks the phat house!! Booyaa! <!--emo&:D--><img src='http://www.unknownworlds.com/forums/html/emoticons/biggrin.gif' border='0' valign='absmiddle' alt='biggrin.gif'><!--endemo-->
This is the kind of innovation that coninually amazes me about the HalfLife mod'ing community. It is the reason Hl is still alive and kicking.
Anyhoo... Excellent work Cagey <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' valign='absmiddle' alt='smile.gif'><!--endemo-->
On another note, removing unused planes probably destroys any hope of the BSP being decompiled back into MAP format... which is a good thing for mappers wanting to keep their work from being altered without their permission.
I've calmed down a bit about everything, having had some time to reflect on why people might have problems believing me. It's helped that people have reported some success using the program so the thread isn't completely a list of people saying I'm full of it. I'll be vindicated eventually anyway, and I guess I can be patient <!--emo&:)--><img src='http://www.unknownworlds.com/forums/html/emoticons/smile.gif' border='0' valign='absmiddle' alt='smile.gif'><!--endemo-->.
As for modifying the <i>compiler</i> to use more than the 32767 planes, that's as simple as
<ul><li>downloading the source to the compiler<li>opening the file "common/bspinfo.h"<li>moving to the line 26, which says "#define MAX_MAP_PLANES 32767"<li>changing that define to a higher number (65535 would be a logical next size)<li>recompiling each tool with the new limit</ul>
It would be more flexible and require less memory if the size were entered from the command line with a default of 32767 so you don't allocate the extra memory unless you need it, which is the benefit behind making a switch like "-maxplanes". If you have the GNU compiler and experience with compiling programs, I invite you to try the change listed above yourself by downloading the source of 1.7 from collective.valve-erc.com. I have asked Merl's permission to make a copy of the tools with this change publicly available, but won't post it without his blessing. <span style='color:green'>EDIT: Merl told me it was fine to post the modification, so I've done so on the third page of this thread.</span>
The reason why nobody has done this before is it breaks the game to have more than 32767 planes--it's a hard limit in the engine, so there's little point in moving over it... unless you're going to move back under, which is a new option. Up till now there hasn't been any point in making the change (in fact, if you read the chunk of the 1.7.1 manual I posted on the first page of the thread, there's a warning about this). There isn't any inherent reason why the compiler itself can't up the limit for internal calculation purposes--it would mean allocating more memory, but aside from that and a fractionally slower hlcsg run there shouldn't be any negative effects on the <i>compile</i> process. In fact, zoner has upped many of the internal limits from the official valve version of the tools packaged with the SDK. Some of these changes:
max brushes went from 4096 to 8192
max map entities went from 1024 to 2048 (another instance where the compiler's limit would be unhealthy for the actual game)
max ent data size went from 128K to 512K
default max texture size went from 2M to 4M (and this is an adjustable limit now)
max lighting size went from 2M to 4M
max size of an entity's value in a key/value pair went from 1024 characters to 4096 characters
My program uses the valve SDK version of bspinfo.c, mathlib.c, cmdlib.c, and scriptlib.c with most of the limits in bspinfo.h bumped to zoner's levels--and the max planes define bumped to 65535 so that it can reduce maps with over 32767 planes once they're created. The optimization process itself is laughably simple, and took about 70 lines of code to write.
My question is this, to kind of sum of whats in this thread so far: The tool you made can not currently be used on anything except a compile BSP right? I can't run it on a .map or .rmf I am working on to optimize planes usage so that I can build more complicated areas into my level without running over the max 32k planes, right?
I see that a fix for a larger amount of maxplanes might be heading our way in the future, but I'm trying to discern if there is anything available to me right NOW to allow me to build a more compilcated level.
Thanks for the good work so far bud, keep it up.