267: FPS quality, part 2: Files and the critical path
matso
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
Master of Patches Join Date: 2002-11-05 Member: 7000Members, Forum Moderators, NS2 Developer, Constellation, NS2 Playtester, Squad Five Blue, Squad Five Silver, Squad Five Gold, Reinforced - Shadow, NS2 Community Developer
The second part of the NS2 frame time quality pretty much comes down to incomplete asset setup.
Essentially, all assets (models, textures, animation, fonts, sounds, materials, shaders etc) should have been loaded and compiled/prepared before you start playing the game so that if you need an asset while the program is on the critical path (ie working on the next frame), it is instantly available.
That's what NS2 is doing during that long Loading/Precaching phase. Or rather "should" be doing.
When looking at hitchy "p_logall" logs in the PerfAnylyzer, I could see that many of the had a of File::Open lingering in the neighbourhood. Unfortunately, there was no way of seeing WHAT file was opened in the profiler.
Once I had engine access, I added a "file critical path" logging - ie, you could tell the filesystem when it was on a critical path so it could log file opens there (there is a lot of files being opened in non-critical paths (texture streaming among other things), so logging all file accesses would be tedious).
After looking at that, it turned out that there was a lot of files being opened in the wrong place, and for lots of different reasons.
- Fonts were opened up on demand, and the FontManager kept track of how many were using a font ... and releasing it when the usage count went to zero.
But sometimes your HUD display pops up an element that is the only users of a font - which means you take a hit loading the font from the filesystem AND compiling it; an average of 5-10ms. Then the element fades out ... and the FontManager promptly releases the font, preparing for another 5-10ms hitch the next time that HUD element needs to be shown.
- Materials are short files ~100bytes or so. They were considered too small to be precached... which is fine if the file is in the filesystem buffers or on an SSD, but if not - 5-20ms hitch for each (http://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics).
- Cinematics is kinda like a movie containing sounds, models (with materials and textures) and a script; there are droves of them in the game, doing everything from waterfalls to railgun splatting.
Turns out that cinematics precached their textures. Nothing else. Which meant that the first time a cinematic was played for you, it would start out by compiling any models that wasn't present already. Starting by loading them from the filesystem... and a cinematic can contain upto 20 models. And models take time to compile... hi, 100+ ms hitch.
As cinematics control stuff like blink effects, blood spray etc ... guess what happens the first time you are in combat - or the first time you meet a new lifeform or marine weapon.
- The GUI system is used for the HUD display, and it's ... weird. All the rest of the lua code is loaded when the client starts, but the GUI code is loaded on demand. So the first time you become a jetpack marine or an Exo or an alien, the gui script is compiled.
Not so bad, really - a one time hitch the first time you become a new lifeform isn't really anything to worry about.
However, it interferes badly with how you precache things in NS2. You do that by doing a compile time statement that you will be using an asset, so as the lua code is being loaded, the NS2 client is being told what assets it needs.
Works well when the lua code is loaded during startup... actually, it ONLY works during that time, as the precaching system does not like being used DURING the game ... then it's not _pre_ caching, really.
So no assets used only by the GUI elements are precached... and there are 422 files in "ui/*" just waiting to hitch your frame.
- The only weirder part than the GUI system is actually a GUI subsystem called GUIViews ..this allows you to paint on a texture, and originally (like, back in the oughties) this was done in its own Flash virtual machine. Adding Flash just to tell you how many bullets were in the rifle was a silly idea, so Flash got kicked out and replaced by a Lua VM.
Unfortunately, it wasn't replaced by using the already existing Lua VM. It was replaced by an individual Lua VM. One for every display. So a standard marine has one Lua VM dedicated to his rifle bullet display and one for his pistol bullet display. Oh, and one for the builder display. And another for the welder, if he has one.
This isn't really as bad as it sounds - Lua VMs are small and cheap. The problem here is that the displays are dynamically created and destroyed - if you drop your rifle, the Lua VM is destroyed. When you pick it up again, a new Lua VM is created - AND loaded up with code ... which is read from the filesystem.
And of course, the Lua code used in these separate lua VM don't even have access to the precaching system (not that they could have used it anyhow ....) so any textures or fonts used in them will of course not have been precached.
So if you wonder why picking up today's first shotgun off the floor can freeze you for 100+ ms, now you know.
Anyhow, most of these things (the most critical stuff) are fixed in 267, resulting in a much smoother experience.
The rest are scheduled for 268.
Ain't gonna be no hitching on CDT's watch.
Essentially, all assets (models, textures, animation, fonts, sounds, materials, shaders etc) should have been loaded and compiled/prepared before you start playing the game so that if you need an asset while the program is on the critical path (ie working on the next frame), it is instantly available.
That's what NS2 is doing during that long Loading/Precaching phase. Or rather "should" be doing.
When looking at hitchy "p_logall" logs in the PerfAnylyzer, I could see that many of the had a of File::Open lingering in the neighbourhood. Unfortunately, there was no way of seeing WHAT file was opened in the profiler.
Once I had engine access, I added a "file critical path" logging - ie, you could tell the filesystem when it was on a critical path so it could log file opens there (there is a lot of files being opened in non-critical paths (texture streaming among other things), so logging all file accesses would be tedious).
After looking at that, it turned out that there was a lot of files being opened in the wrong place, and for lots of different reasons.
- Fonts were opened up on demand, and the FontManager kept track of how many were using a font ... and releasing it when the usage count went to zero.
But sometimes your HUD display pops up an element that is the only users of a font - which means you take a hit loading the font from the filesystem AND compiling it; an average of 5-10ms. Then the element fades out ... and the FontManager promptly releases the font, preparing for another 5-10ms hitch the next time that HUD element needs to be shown.
- Materials are short files ~100bytes or so. They were considered too small to be precached... which is fine if the file is in the filesystem buffers or on an SSD, but if not - 5-20ms hitch for each (http://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics).
- Cinematics is kinda like a movie containing sounds, models (with materials and textures) and a script; there are droves of them in the game, doing everything from waterfalls to railgun splatting.
Turns out that cinematics precached their textures. Nothing else. Which meant that the first time a cinematic was played for you, it would start out by compiling any models that wasn't present already. Starting by loading them from the filesystem... and a cinematic can contain upto 20 models. And models take time to compile... hi, 100+ ms hitch.
As cinematics control stuff like blink effects, blood spray etc ... guess what happens the first time you are in combat - or the first time you meet a new lifeform or marine weapon.
- The GUI system is used for the HUD display, and it's ... weird. All the rest of the lua code is loaded when the client starts, but the GUI code is loaded on demand. So the first time you become a jetpack marine or an Exo or an alien, the gui script is compiled.
Not so bad, really - a one time hitch the first time you become a new lifeform isn't really anything to worry about.
However, it interferes badly with how you precache things in NS2. You do that by doing a compile time statement that you will be using an asset, so as the lua code is being loaded, the NS2 client is being told what assets it needs.
Works well when the lua code is loaded during startup... actually, it ONLY works during that time, as the precaching system does not like being used DURING the game ... then it's not _pre_ caching, really.
So no assets used only by the GUI elements are precached... and there are 422 files in "ui/*" just waiting to hitch your frame.
- The only weirder part than the GUI system is actually a GUI subsystem called GUIViews ..this allows you to paint on a texture, and originally (like, back in the oughties) this was done in its own Flash virtual machine. Adding Flash just to tell you how many bullets were in the rifle was a silly idea, so Flash got kicked out and replaced by a Lua VM.
Unfortunately, it wasn't replaced by using the already existing Lua VM. It was replaced by an individual Lua VM. One for every display. So a standard marine has one Lua VM dedicated to his rifle bullet display and one for his pistol bullet display. Oh, and one for the builder display. And another for the welder, if he has one.
This isn't really as bad as it sounds - Lua VMs are small and cheap. The problem here is that the displays are dynamically created and destroyed - if you drop your rifle, the Lua VM is destroyed. When you pick it up again, a new Lua VM is created - AND loaded up with code ... which is read from the filesystem.
And of course, the Lua code used in these separate lua VM don't even have access to the precaching system (not that they could have used it anyhow ....) so any textures or fonts used in them will of course not have been precached.
So if you wonder why picking up today's first shotgun off the floor can freeze you for 100+ ms, now you know.
Anyhow, most of these things (the most critical stuff) are fixed in 267, resulting in a much smoother experience.
The rest are scheduled for 268.
Ain't gonna be no hitching on CDT's watch.
Comments
You really can't look at it that way. Many of the issues matso mentions, took a long long time to debug and find. Some of it was not even known to be an problem until recently. That is how software development works. UWE did an amazing work on NS2, and now the CDT can build on that great foundation. Some of the work being done now is also very time intensive for very little gain, something UWE could financially not always justify. I see changes from people like samusdroid correcting small typos in some obscure texture on a crate used in a hidden place in one map. This is amazing work but makes financially no sense. So lets not critizise UWE but be happy for all the changes that can be made now.
Also, only judge those changes once you actually play with them. They are not the silver bullet to performance problems. Yes, some computers will benefit from them, but others wont and lot of work/research will have to be done to help them also.
I've been having micro-hitches for a very long time and this is the exact fix that is needed for myself and quite a few other players. The fact that it took this long to get to the bottom of it is a problem. I know that work can always be improved, and part of that is that there is less and less obvious things to be fixed. The micro-hitching though? Too common for it to be found now. It is not like it is a small typo in this case.
Even still, I should have wrote the original post in a better manner. "This should not have happened", should be, "this should have been fixed way before.". I'm trying to think of a more common issue that harms the game that has been around for as long. I know IronHorse can find something.
How much it will help almost doesn't matter - any improvement is a good thing. Stuttering is something that pretty much anyone on any system can experience in NS2, so any step to address this is brilliant
Thank you @matso for the very clear information on these posts! More!
i would rather describe him as hot!
Also I would not bugger uwe to hard about not finding these bugs sooner. Proving there was a bug for many was surely hard enough, and thats usually step one in fixing a bug.. right?
After reading through this, reading each of the 'ready for 267' posts in the CDT Public Tracker, I can easily say this patch should greatly improve my overall enjoyment of the game.
Now I know this is no silver bullet and some things will have to come later, but anything that can reduce the amount of hitching NS2 does with my old Radeon HD 6770 GPU and 7200RPM HDD will probably cause me to play it a lot more than I have been.
The "Smooth PhaseGate exit" fix in the Public Tracker has me excited too, I've had to play on the lowest settings because how bad the hitching can get overall. (To compensate for lost frames.)
:P
With the tape deck!
Had so much patience with those.. Now if software takes longer then 10 seconds to load I start to rage
How do you cope with battlefield games? Yeesh.. takes me like 5 minutes to load
(besides the atrocious lensflare / visual obstruction/ low skill ceiling gameplay)
Easy, Don't play BF games!! I got BF BC3 free, and I loaded it up once. then deleted it. NS2 so much better.
2 GB? 3 GB?
that would be Mendasp.
After the update... it should probably be anything around 1GB being sufficient for high textures I hope. Proper streaming should allow for almost any resolution of textures to be viewed up close... it is just their quality while moving around or in the distance that should be compromised.
Higher amounts of VRAM should allow for more temporal stability in texture quality IMO, not better textures when viewed up close (unless the gulf in texture RAM is huge like 6GB)
1.6-1.8 Gb allocated is the about the high limit with high textures; depends a bit on the map.
So a 2Gb card do not have any problem with running high textures.
Medium textures have half the dimension as high (you drop one MipMap level), so they take up 1/4 of the high textures (and low has 1/16th of high).
As you have some (say 100Mb) of other stuff, that means medium textures hits about 600-700Mb and low 300Mb, about.
And yea, the texture streamer actually works well enough that you can run on high textures with a 1-1.5Gb card (if you have less than that, used medium textures).
If you have a dual core, the extra work from the streamer might lower your fps by 1-2% though.
A privilige reserved for the really usefull or really useless forum members.