Is There A Plan To Speed Up G8 (and future generations) Loading Times?
Hi everyone! :D
As a content creator, I need to keep many morphs installed, and they dramatically lengthen both the loading and the closing time of Daz studio, to the point they sometimes take MOST of the time I need to create content.
So no, I won't create folders for my morphs and turn them on and off based on what I'm doing, that's not convenient for my workflow. And it's not what this thread is about.
I've already done everything that's in my power to reduce these loading and closing times (uninstalled what I don't need, upgraded to NVMe SSD only library, upgraded my CPU/RAM), and I've actually made a guide about it a while ago, that helped me reduce the loading times by almost 80% (https://www.daz3d.com/forums/discussion/411446/why-does-it-take-so-long-to-load-figures-solved-with-guide/).
Although, they're still pretty long.
So, my question is: "Are you making this a priority, and trying to reduce the lengthening of loading and closing times due to G8 morphs installed?"
Thanks!
Comments
I think (hope?) these problems are caused by the massively outdated framework they're using. Qt4 is over fifteen years old by now, and it reached its end-of-life in 2015. I don't think a File I/O system developed for Windows XP can cope with the strain.
But the release notes say they're moving to a newer Qt version soon, so hopefully we'll see some performance gains.
I definitely think Daz should prioritize this over certain other ventures, mentioned elsewhere.
It is all relative. Yes, G8 figures can be slow, but when I started an "All the Vickies" idea, I was reminded just how long it takes to load V4.
That's good news, I hope that'll fix the problem and come soon!
Well, I don't know how slow it is in relative terms, but it's surely crazy slow in absolute terms!
Sometimes it takes more to load another scene than to render it. And Iray renders are not the fastest!
The timeline is this;
https://www.daz3d.com/forums/discussion/451486/daz-studio-macos-big-sur-compatibility#latest
The update will also render some Plugins extinct for all users (both MAC and Windows) as there is nobody to update them anymore
I wouldn't bet on it. What is fundamentally slow is 1. load json data, 2. create erc formulas. Neither are really related to qt4 and both are not trivial to make it multithreaded. Loading formulas are required for a lot of expressions and poses to work correctly.
Loading data is petty trivial to multithread because nothing prevents processing multiple files in parallel.
Loading formulas can be omitted entirely if that property is not in use.
to speed things up even more UI could be made responsive before every morph is loaded and loading could just continue later gradually updating the scene into the final version.
In fact, Daz already has that functionality because it can add extra morphs later so there is nothing that would prevent more loading processes to be put into the background.
But there is a strange general Daz policy to lock up UI in all situations when this is even entirely unnecessary. like during the rendering of D-force simulation it would be totally ok to continue working on the scene while other things are happening in the background.
Yeah, the WP Guru did an article about the issue almost a year ago now. He speculated that since Qt4's scripting engine was deprecated, Daz Studio's own scripting system will break too.
I think it is related. Cross-platform frameworks typically handle file I/O, so app developers don't have to deal with the operating system directly. And a lot of the loading time seems to come from searching through different directories for files. If you split your morphs up between different drives, loading takes much longer. So it is heavily dependent on Daz interfacing with the OS and loading files (presumably) through Qt.
They do have the Cache file that is supposed to speed up the process as the individual morph files don't need to be found and read separately, but the information inside the Cache file is still the same as what can be found inside the morphfiles, except for the morph delta's.
The explanation has been that when the morph data has been read, the dials are created along with any and all dependencies to other morph dials, and this is THE process that takes time.
To me, the obvious solution would be that the Cache file would contain the ready processed information, so it wouldn't need processing every single time you load a figure.
While that sounds like it should be correct, when I consolidated my content directory onto my new external hard drive, I found the loading time for G8 went down. So I'm not convinced the cache file works that way. Or, at least, I'm not convinced it works well.
Judging by the countless posts and discussions on the issue, it doesn't seem to help the process at all, which is logical as the only thing it is supposed to bypass is the reading of individual files in different folders, which is not that much of an issue with modern SSD's - It does nothing to what is said to take the most time in loading a figure.
How about just loading the sliders with labels for the morphs/expressions/whatever, etc,etc and skip the formulas and whatnot.
Load basic character and/or character preset.
Add a load button to each slider. User wants to load an expression, then user clicks the "load" button to activate that slider and pop up a notice if it needs others activated an offers to add them to the "load it" list.
Basically load/unload them on demand, just in a more user friendly way than asking during character load time to decide from 1000's of options what to choose up front(depending how many morph, expression, etc,etc are installed).
Would something like this work?
Discobob & Zev0 already figured it out with "Fit Control":
We don't need ten million sliders cluttering up the property pane, especially when they're not labelled with the product name. Just load them on a product-by-product basis, please.
Fit Control is the only product I have that loads it's morphs in such a way.Everything else preloads with every character every time I load one.Which slows down the load times.
Maybe have everyone switch to the same technique fit control uses then?
What about character presets? I would rather they be in a nice tidy list rather than going directory diving to try and locate them so I can dial in a bit of this one and that one.I use them much like condiments.
Think along the lines of "Oh there's Girl 8, I'll load that one and add 15% of her to this one" as you scroll down the list. The others I don't want to use at the time I can just scroll past and not load.
I agree, sorting the lists by product name would be fantastic.
and is probably not worth doing as it's not the reading that is slow
No it can't, a formula is stored with one of the two morphs so you can't assume that every link is referenced in the file for the morph that is being set and simply load it when it goes non-zero and trace through the links from that. And yes, it does need to have the ability to store the ERC with either controlled morph or controller since the two may be by different people so it isn't possible to have a creator of a downstream morph (say) update the upstream morph's data.
Which would have the potential to haev the user make changes for which the ramifications were not yet established, which would add another round of processing before the load was complete, which would slow tings down further.
Yes, fresh links can be established as needed - but any saving on loading would then add more overhead on future operations (see above) so hardly the royal road to improved performance overall.
So what, the dForce simulation runs in the background while the user carries on posing so that the completed dForce simulation is for the wrong pose/animation? Or would the UI be only partially frozen, that sounds like a good way to frustrate and confuse users (and a good way to lay booby-traps for conflicting data,especially considering third-party plug-ins).
It appears that you do not fully understand what DS is doing, and this is leading you up the garden path.
Not at all the same. Load Morphs applies the morphs to the base figure, which projects them into the fitted item; remove then removes the unwanted moprhs from the fitted item. Load heer is not the same as loading the morphs into the base figure.
Sure, on a technical level, they're different. But I was referring to my desired user experience.
If I want to use, say, Zev0's aging morphs, I'd much rather prefer to just select it like I do any product and have them loaded onto the figure on demand.
What I was trying to say I do't expect the load time to improve from upgrading to qt5 because file I/O part is not the slow part and is more limited by os and hardware. It is converting from text to data and using the data to create formula that is slow, and i am starting to think even the text to data part is not that slow.
Wtih regard to cache and spreading files into multiple drives, Daz will save a cache of all the formulas it find so it doesnt have to open all the small files everytime you load a character, just the first time after you add/remove some morph. But Daz also search through all the directories to check the files have not been updated (i think based on timestamp). This part is affected by your hardware. What I have observed is if the drive is busy doing other data transfer, loading slows down signifcantly.
Daz definitly should work to improve the load process, but I don't expect it to come from the work to upgrade the framework, rather from a dedicated effort with lot of developer hours.
It would be very strange if anyone besides Daz developers had any understanding about that. without any access to source code or any information besides what can be found on this forum in the posts like this. So obviously I can just make guesses.
But the situation is pretty bad and you cannot just ignore it and pretend that everything is fine.
Everything is getting slower every day and installing any new product(especially expression packs) becomes a serious choice because it will slow down things even more.
Long load times are not even the main problem anymore, because everything is getting very sluggish even selecting a figure requires few seconds, IK manipulations take over 10 seconds just to be activated and work incredibly slow.
If nothing will be done Daz is going to become unusable.
And what is so horrible about the fact that this will happen? If you see that it looks wrong just run it again. It is far more frustrating to see that simulation went totally wrong anyway.
This is the general issue with Daz were in an attempt to prevent some small confusion extremely serious limitations are introduced.
when Dforce is working on one character I could work on another at that time or move the viewport around to see if it is doing its job properly without the need to stop it in the middle check if everything went right then redo it all again .
maybe Dforce can even run in the background and respond to the user adjusting the pose, rather than relying on memorized poses
Well, I agree it would be nice and useful to be able to move the Viewport as the simulation was running. However, I think anything beyond that would be potentially disastrous (even if it also might be nice and/or useful). You can always make a feature request https://www.daz3d.com/help
A very well informed opinion, that. Without knowing anything about DS's architecture, and even the C++ SDK does little to enlighten us, there's just no way to know for sure and all we can do is make logical inferences from what we do know.
I think Daz's framework is very good in many ways, but I simply cannot make up a viable excuse to justify a framework that loads things it doesn't need, or allows those unneeded things to sometimes corrupt a figure that doesn't even reference it. When this happens in software development, more often than not it's because the app has outgrown its original design, but can't be upgraded because of a third party tool that isn't available, backwards compatibility, maybe even in-house code where the original author has moved on and no one else understands, or it would just take too much effort to update.
All of these things are described in the term "technical debt", which is analogous to buying expensive things you really can't afford using a credit card. The bill will eventually come due, and the longer you put off paying it, the more it's going to cost. From other things I've observed as an outsider, it really seems like Daz has an enormous amount of technical debt. In the fondest of my fond dreams, DS5 is a clean slate and I've allowed myself to be convinced that that's why it is taking so long.
But really, as @onix points out, there's just no way for any of us to know the what, why or when of it.
How can you say you can't know the details, then assert that DS is doing something it doesn't need to do - especially here, where the reason for loading all morphs (to get their relationships) has been explained repeatedly?
You did not really explain that much WHY this is required. why do we need those relationships and can't just live without them
The reason why we can say that it is not required, is that we can just temporarily hide those unnecessary morphs, and everything works, so all that Daz has to do is just for the bare minimum is to replicate that hiding process and simply not to load things that are not in use. If only we had the ability to load that hidden morphs later it would be near perfect solution. It could even solve the problem that you cannot install new morphs on the already loaded figure, you have to reload the scene. On-demand loading would solve that as well.
Or maybe we just don't even know what is the real purpose of why we need those relationships in the first place. And maybe we can just turn off that "important" feature because it is not needed entirely?
How do you even explain the situation when some rogue morph file, which is not even referenced by the used anywhere, is messing up the figure?
That looks more like a bug than a feature. Such files should not have any ability to get injected into the model.
I can only guess that the reason why this was done was to allow some kind of plugin style morphs where you install some corrective stuff which gets turned on without user consent just by the mere fact that it exists.
But by all sane logic that should never happen. Morphs should be turned on only if the user decides to do so or some other morph specifically references them. Installing third-party models should not in any way affect the base model without user consent.
It is even a serious security issue because someone can deliberately or accidentally ruin someone's else product and make it behave in a very strange way
At least in my understanding of how things should work is this:
1 read and parse every morph file just to build a list of sliders for the user to choose but don't read anything further don't build any relationships just make a list of what is available
2 read the scene file to gets a list of what morphs are non-zero there.
3 check if those non-zero morph files do exist and if they exist then parse them look what kind of morphs they are referencing them parse those morphs and traverse the morph tree that way. This can be easily multithreaded as well because top morphs do not need to know anything about each other relationships. everything loads totally independent from each other.
Everything that was unreferenced is left unparsed and nothing about those morphs besides their names is even stored in memory.
The relationships are needed because when you dial MorphSoandso it may trigger JCMSuchandsuch, if BendThatone is also set. Without knowing the relationships DS won't know which other morphs are needed, and which other morphs they need, and so on. As I think I said above, these links are stored in one of the related properties' files - but it can be either, so there's no guarantee that reading the file for MorphSoandso will help (and because links may be, probably will be for JCMs, between properties from different sources there's no way to safely write the relationship details to both files.
If you hide morphs how do you know you are dealing with all the associated files? If nothign else you are likely to leave a lot of dangling dependencies - and of course the issue is much worse post-loading the morphs since again there may well be relationshops thata ren't listed in the product files that nevertheless need to be established.
I can see why the OP, as a content creator, needs so many characters and their morphs installed. My load times, by comparison, are very short - <10 seconds for a G8 character. I assume that is because I have only around 20 characters in my library. I do have a lot of morph packs though - mosty from Zev0. So maybe the solution is to buy less pre-made stuff and dial-in more shapes using the morph packs? I know that some people have hundreds of characters and that they have excruciatingly long load times.
Because I made a distinction between things we can observe externally without knowing the internals, and the things we cannot know. Many people have observed it empirically by having their character distorted by an errant morph that the scene does not even explicitly use.
And just because a particular has been explained, repeatedly as you say, does not mean that the behaviour represents a good design. My point is that a design that requires loading morphs that are not even used, for whatever reason, is not a good design. It may have been an acceptable at some point, but it didn't scale and today it is not. I don't think the people suffering from its artefacts are very much consoled by the "explanation"; my doctor telling me "Well, you see, your head hurts because your hair appears to be on fire" doesn't make it any less painful.
Let's stop making apologies for a bad design that needs to be re-thought for DS to keep moving forward.
+1000
@onix you said it better than I did. In no other area of computing would this be acceptable. This is the equivalent of Windows 3.0 and no one wants to be the equivalent of Windows 3.0
It's a bad design. Period.