The cause of two FPS killers: day number/framerate and screenshots
Belgarel
Join Date: 2017-07-03 Member: 231570Members, Subnautica Developer
Join Date: 2017-07-03 Member: 231570Members, Subnautica Developer
This is for the current stable version, if these have been fixed already please tell me, otherwise ensure they make it to the developers. I've identified the two main causes of the gradual FPS death in the game.
The first is likely a mathematical precision error when updating the game time and should be trivial to fix. It's the cause of the 'stuck moon' and 'fabricator slow' and 'fabricator only makes progress when taking a screenshot' bugs people report. If the number of elapsed game days is high and/or the interval between frames is short, when the game decides how much to advance game time by per frame it might get rounded to 0 and no time passes. To demonstrate this, here is a save game where I've done nothing but start the game and use 'daynightspeed 100' to advance the clock to day 697, subnautica day 697.zip. At 60fps or higher, the moon is stuck in the sky and the fabricator is very slow. At 30fps (use rivatuner or burn CPU like via the screenshot key) you'll see it moving again. People who play at high framerates will run into this bug sooner than others.
The second is that the number of screenshots in the screenshots folder decreases FPS. It seems like every frame the game is checking the screenshots, even in a fresh game without picture frames. With 250 screenshots in my folder, I went from 144fps to around 30fps. If you have a less powerful system or are on a magnetic drive you might see a large impact sooner. You can test this yourself by adding screenshots until you notice the performance drop, you don't even need to restart the game, you'll see it change live. The workaround for this is to just delete your screenshots folder.
For the first issue, the only workaround I have is to edit the save game to roll the clock back via editing the time in AtmosphereDirector, but the time is seemingly used elsewhere in the player data to track exhaustion so I don't feel safe doing it with only partial knowledge of the save data. Please fix that bug as it's a game killer.
The first is likely a mathematical precision error when updating the game time and should be trivial to fix. It's the cause of the 'stuck moon' and 'fabricator slow' and 'fabricator only makes progress when taking a screenshot' bugs people report. If the number of elapsed game days is high and/or the interval between frames is short, when the game decides how much to advance game time by per frame it might get rounded to 0 and no time passes. To demonstrate this, here is a save game where I've done nothing but start the game and use 'daynightspeed 100' to advance the clock to day 697, subnautica day 697.zip. At 60fps or higher, the moon is stuck in the sky and the fabricator is very slow. At 30fps (use rivatuner or burn CPU like via the screenshot key) you'll see it moving again. People who play at high framerates will run into this bug sooner than others.
The second is that the number of screenshots in the screenshots folder decreases FPS. It seems like every frame the game is checking the screenshots, even in a fresh game without picture frames. With 250 screenshots in my folder, I went from 144fps to around 30fps. If you have a less powerful system or are on a magnetic drive you might see a large impact sooner. You can test this yourself by adding screenshots until you notice the performance drop, you don't even need to restart the game, you'll see it change live. The workaround for this is to just delete your screenshots folder.
For the first issue, the only workaround I have is to edit the save game to roll the clock back via editing the time in AtmosphereDirector, but the time is seemingly used elsewhere in the player data to track exhaustion so I don't feel safe doing it with only partial knowledge of the save data. Please fix that bug as it's a game killer.
Comments
I made an example of the issue in python for you that does the same math as your DayNightCycle class to demonstrate the problem. After 109 in-game days, at 144 fps time will stop advancing. After 437 days, time will stop advancing at 60 fps. There are a lot of artifacts before these times too due to the precision (e.g. time will appear to move too fast and the fabricator will finish slightly before the sound effect).
#!/usr/bin/python from numpy import float32 from datetime import datetime # A game day is 1200 seconds. Initialize timePassed to what 09:36am in # game day seconds would be (480). dateOrigin = datetime(2287, 5, 7, 9, 36, 0) timePassed = float32(1200) * (float32(3600) * float32(dateOrigin.hour) + float32(60) * float32(dateOrigin.minute) + float32(dateOrigin.second)) / float32(86400) # Frame rate we're rendering at, and the Time.deltaTime Unity will give # us. deltaTime will be 0.006944. fps = 144 deltaTime = float32(1./fps) # Show what updating by deltaTime at the beginning of the game actually # changes timePassed by. Actual change will be 0.006958. print 'timePassed: {}, deltaTime: {}, timePassed + deltaTime: {}, actual change: {}'.format(timePassed, deltaTime, timePassed + deltaTime, (timePassed + deltaTime) - timePassed) # Advance 109 game days into the future. timePassed += float32(109 * 1200) # Try the same update. This will give an actual change of 0. This is # why time appears to freeze up in long running games and happens # sooner the higher your FPS. print 'timePassed: {}, deltaTime: {}, timePassed + deltaTime: {}, actual change: {}'.format(timePassed, deltaTime, timePassed + deltaTime, (timePassed + deltaTime) - timePassed)Output:
EDIT: 'Fixed' by changing them to single quotes.
The solution you guys should probably implement is just switching timePassed to a double as well as changing anything that references it to use doubles as well (CrafterLogic's progress accessor as one example). It'd be pretty easy to do, probably about an hour, and is relatively low impact.
I went with a less ideal solution due to lack of a decompiler that works 100% with your Unity version - changing a lot of code to do double precision math via editing the DLL at the IL level seemed a bit too Maida for me. I create a double precision non-serialized preciseTimePassed variable and do the deltaTime math on that, then cast it to a float and save that as timePassed. That means that while some frames still see zero time passing (since timePassed is still a float), time will continue at the correct rate over multiple frames. So the moon kinda jerks around in the sky like it's on dialup but it works - I played several hours on my save game that had been killed by this bug with no problems and it survived saves/loads. With this method I only had to change small parts of DayNightCycle and a couple other places where timePassed is assigned and didn't have to worry about save game compatibility.
If someone's interested in my temporary fix, it's here: fix-subnautica-stuck-game-time.patch. Update() looks like a bigger change than it is as I just cut+paste the whole modified method to save time. Not posting the patched Assembly-CSharp.dll itself as I figure you'd not want me to. No warranties, back up your saves, etc..
Of course, interwebs has plenty more and better guides for these, but those few should already get one much further than just using float and using the first operations that seem to match the wanted functionality.
And then finally, one does not have to move things with "delta-per-frame"; instead recalculate the position always based on "since start of time" (instead of "since last frame"). However, this has its own floating-point monsters, but they are often easier to understand than the accumulating rounding error.
(EDIT: above applies to not only "moving things" but "advancing time", too.)
(My background was with calculating money values with floats, in a pinch, when decimal based data-formats/routines were not available. Have to do all kinds of tricks to ensure accurate results (to the 0.01) in all kinds of cases. Extra trouble from the fact the (binary) floating-point formats suck in handling base-10/decimal based divisors/fractions.)
I guess obs (open bradcaster software) is quite known.
And i also know for a fact that the timings used in there are accurate to up to 200 years. I daily lurk in the dev. channel of that oss and the main developer actually stated that. So perhaps it would be interresting to have a look at that code on github.
The increased precision is fine for what they're using it for and a simple fix.
It's not an order of operations issue. Look at my python example. They are unable to represent one frame later than 131280 seconds from the beginning of the game. The correct time is roughly 131280.007 but the two closest values they can represent with a 32 bit float are 131280.000 and 131280.020. They can't even store the current time at that precision let alone manipulate it.
What complicates this is Subnautica has a mechanic to speed up / slow down game time and it's also used to implement sleeping. So a "time since start" interval might contain several periods where game time moved at different speeds. They'd need to keep track of the amount to shift game time by to keep that feature working, and to keep that from having the same add-per-frame precision problem it would need to measure the time since they last changed the game time speed. It's complicated, wouldn't have any real effect on the game, they'd still need to increase precision anyway (see above), and they seem to be behind, so it's not worth it.
No, using floating-point format (even double precision) is not "fine" (unless using the sarcastic tone of "fine, whatever, I give up") when using it as part of math where the time (or whatever parameter) changes linearly and is supposed to keep its accuracy over the whole valid range. (These "it's fine" attitudes without actually thinking things through have even killed people, although I think we're safe in this case
Oh well, if they keep in that style of using high values, and want to stay with floating-point stuff, I'd also change it to double. But if changing it, why not use the opportunity to change it properly, e.g. to 32 or 64 bit fixed point, which would have the same accuracy no matter which value it has. 32 bit fixed point would be enough for 3000+ game days at 0.001 sec resolution. Sounds big enough? Nope. I'd go with 64-bit one, just in case someone tries to make a few month marathon of subnautica playing (what humans can and will do surprise time after time). Unless there is some hard limit in the game which always enforces "game over" or "win" before certain number of days.
I had also made the assumption that the game would keep track of those sleeps etc. time rate adjustments in a sane way. (I.e. the delta time calculations would be given, as inputs, the current time rate factor and/or additional "external" shifts as needed). Naturally that part of my suggestions goes down the drain then.
Note I did not say order of operations (or other suggested points) would be the problem (and thus changes to them the fix); I meant that such changes would make a better fix if continuing in the floating-point domain (whatever finally is needed in addition)... Not that better one would be necessarily required, it is not like few extra double-ops per frame is any kind of load to worry about. Brute force (with consideration) is at times a valid solution, even if not an elegant one :P Brute force with ignorance tends to often bite back.
Your python example did show the problem of adding small value to large value, but did not let me know what the actual inputs and output of that part of the game code are, or can it keep own state up somewhere (the accumulation of rounding error). Thus, my suggestions were more like general ideas often used when trying to keep floating point errors as small as possible or trying to squeeze more out of floating-point formats when there are no other more appropriate solutions. (In this case, there are.)
In any case, I'll rather have a simple fix (just slap double precision all over the place) than no fix at all, if a proper solution is too much. ("Fine, whatever, I give up"
You have an irrational fear of floating points. For the game speed, no one's going to notice if it's 1% off. With a 32 bit float, 1% error at 144hz happens after the first 2**12 seconds = 34 minutes. With a 64 bit float, it's 2**40 = 35,000 years. That clearly works fine as the error fits the precision they need.
Put another way, Windows by default uses SNTP and isn't drift-corrected. The clock at the lowest level of that OS drifting by up to 100us/sec is normal. At the 5 hour mark in Subnautica, the inaccuracy from use of double in that addition would also be about 100us. So you're concerned about losing precision that wasn't there to begin with.
Time in Unity is internally a double and will have already lost the precision you're hoping to maintain. Converting the code to custom timing would be a lot of work for a difference imperceptible to players.
No irrational fear here, just the long-time experience on bumping almost weekly on bugs caused by developers not following good programming practices. Occasionally my own bugs, me not always following them good practices
Floating-point values have their own "best fit" uses, too. This isn't one. Money values is another where they should never be used (except perhaps in economic simulations and such where the input values and functions themselves are already inaccurate). And even time can be converted into floating-point during various calculations that derive from time, but the variables that hold and count the time should not be.
These accuracy things have been solved (i.e. found out what is the good practice) decades ago, yet these keep coming up all the time. In some cases there are benefits of using the floating-point stuff, like in this case it is just convenient due to Unity already using it. But other times it fails, (like it did in this case). The usual solution, which you are suggesting, is indeed typically the brute force and ignorance, "just add enough bits". Which typically moves the problem further away, sometimes far enough that it won't matter. (In this case, the bare mantissa of double-precision float has enough precision, the exponent part could be ignored, but then it would be used just as an integer.)
But if you never learn what is the right solution, you just keep repeating the same mistake everywhere, and end up using it in a case where it does fail. E.g. using floats for distances for all purposes (not just the rendering and local physics), and once visual glitches appear, you just bump it to double, problem solved... and then wonder why year later people start to cry about frame rates dropping to a fraction of what they were before. Etc. etc.
I wasn't trying to get the precision better than what double can give. My goal was/is to get good enough precision and value range while also following a good programming practice.
It's a best fit for Unity development. Some Javascript (UnityScript) often ends up in Unity projects and it doesn't support integer types yet will need to agree with the C# code on key things like time.
Floating point didn't fail, the Subnautica guys just used a type that is too small for the precision they need. It's a common mistake when developing with Unity.
I'm not sure how checking the required precision and ensuring it fits a type is ignorant brute forcing. It's the same concept as significant figures in lab work.
Good programming practice is not turning a 1 hour fix into a 1 week fix with added complexity and code in order to make an imperceptible improvement.
You did check, many programmers do not. And even if they do check the requirements, they show then ignorance in calculation efficiency etc. (In this particular case we're lucky enough that the efficiency can be ignored.)
Indeed significant figures in lab work are just the same.. and those lab calculations also handle time as just one more parameter with equal effect in errors; they can do so because the reactions or physics do not usually suddenly turn into bigger steps (or stop completely). I think I already mentioned there are cases where time can also be handled with floating-point.
Using your phrasing: Handling time with absolute accuracy is the same concept as using all figures in calculations in combination with money.
(Try using even double-precision time in calculations related with money, and one late evening near the end of a year you'll curse about the missing $0.01 that doesn't let the bookkeeping get through and few people will have to work long days over the Christmas holidays just to try and find where the heck among the few hundred million rows of data it went wrong and why it went wrong.)
This might be now going to either semantics and/or "politics", but as far as I (and my colleagues) understand the term "Good programming practices":
Good programming practice has nothing to do with how long it takes to make (though following the good practices often helps making the best solution pretty much as quick as the other solutions). It does have a bit to do with avoiding complexity, though (that is, sometimes the "best" solution in operational way is just too complex for general use). Also, it has nothing to do whether the effect in the result is imperceptible or not. The point of learning those good practices is exactly to know how to make it (easily) work the best on the first try, and also to cause less issues later. It helps in the long run, and often also sooner.
The coding time and complexity of my suggestions depend on the circumstances. In the simpler cases they can take just as long to code and are just as simple to understand as floats (all one needs to add is one float conversion and a scaling operation in the end). In more difficult cases (where the surrounding environment already causes the need for complexity) it can add anywhere between 10-50% more code, but usually not much more complexity. I do admit that in some special cases the complexity can step up quite a bit, but even in the worst case that I have met, a 5 minute double-implementation turned into about 30-minute thing with 7 lines of comments to explain it to others; I had to move a division from deep in a common function to be the last operation, so the math ended up looking unrecognizable from the usual formulas, and the comments just explained the math for others (and for me years later).
This case would have been one of those simple ones, if there had not been that Unity/other already existing float/double-time parameters.
But, I think we have derailed this topic quite far from the actual issue, into the realm of general programming :P Perhaps we should have own thread in somewhere else (seems Subnautica forums don't have a "anything else" area).
It does, it's just pretty well hidden from the Subnautica-facing side:
https://forums.unknownworlds.com/ <- from there you can get to "Other", "Off-Topic", etc etc as well as forums for the other games UWE has developed (Natural Selection 2, go check it out if you haven't).
As i wrote in my own thread i will test your workaround. But for now: well done mate!
Yeah would be better.
And, anyone with better connections to the devs should give the links to our both threads to the devs. I gave it up.
@EkUl @Belgarel - Future reference, you can ping the devs on their Subnautica Discord server (see the link in Obraxis' signature or {buried somewhere} in my signature). Try to avoid doing it too much, as they seem to be quite busy a lot of the time, but for things like this that haven't seemed to make it to their attention after quite a while, it's better than it never getting noticed.
EDIT: Also, the screenshot bug is here: https://trello.com/c/HyZ45dFS/6543-fix-screenshot-manager-performance
So... perhaps another reason why the day/night cycle should be reduced and have the time cycle slowed down? Not only is it immersion breaking to have many game days spin by in just an hour real-time, but it also kills the FPS after so many game days have passed!
Please, consider reducing the game day/night cycle or at least let us customize it when we start a new game. And before anyone mentions it, yes console command DAYNIGHTSPEED [#] lets me slow down time, but that has the unfortunate benefit of messing up how other time-related effects happen in the game. Plus... I shouldn't have to cheat just to get a more realistic gameplay experience, really.