Loading...
> >

What can we do to improve the long load times of characters?

13

Comments

  • PDSmithPDSmith Posts: 712
    edited February 2021

    @Zilvergrafix  I did the same method you used, then tried something else and found it works just as well and just as fast.

    I right click on the individual morph folders, and in the parameters section I change the folder to hide, then apply.  all folders and files are now hidden. (how many finger nail morphs do we need? )

    Now to the test...loading characters don't error out, but the catch is you can't change those morphs on characters since Studio appears to not know what to do with the files since it can't see the file. From what I've seen here in the forum that is due to the DUF tells Studio where files are located during the load process but that's it.

     

    The one folder I do not hide is the DAZ3d folder.

     

    Heck in one test, I even hid the entire contents of the morph folder but the DAZ3d forlder and G8F and G3F each loaded in roughly 4 seconds and all my premade figures loaded in 8 or less seconds. I was originally up to 45 seconds for g3f load times, and 1 minute 34 seconds for g8f.  and that's with nothing loaded, just the base figure. Load times with created figures would take up to 3 minutes.

     

    Garage logic lesson learned here...

    Unhide folders to make character.

    Hide folders to later use them.

     

    Late edit: This all works fine and great but you need to make sure you've turned on the option in Windows Explorer to see hidden files.  

     

     

     

     

     

     

    Post edited by PDSmith on
  • grimulkan_9cfbd329bcgrimulkan_9cfbd329bc Posts: 63
    edited February 2021

    volpler11 said:

    Did some more poking. 

    The cache file is basically most of the files in "data\DAZ 3D\Genesis 8\Female\Morphs\...." mashed together with one important exception. The bulk of the data in those files are of two types: morph vertex delta and formula. The important exception is that morph vertex deltas are not included in the cache file and is loaded on demand when you change the slider. Formula data still make into the cache and can slow things down significantly. 

    This lead to one important observation: the load time is proportional to the number of files in the morph folder but not necessarily file size. If the file contain large amount of delta that is not included in the cache it will be fast to load. On the other hand if the file contain lots of formulas it will be very slow to load.

    Edit:Also the morph file could be compressed which would make it smaller than it actually are.

    This is gold (as is your thread)! Thanks.

    When I switch between morph folder setups using my tool, I don't really see a lot of time being spent re-creating the cache. Maybe it's because the morphs are still expected to "be there", since they're just hidden, and the cache isn't modified? So maybe DAZ just doesn't bother loading the vertex data or truncates formula calculation if the file is hidden. Or maybe cache creation is only like 20% of the total load time so I didn't notice.

    But your caching explanation explains why my 2nd G8F load in my original test loaded faster than the first. Also, I was parsing the wrong file (log) to try and figure load times, will try again with the cache file. Did someone write a tool for this once?

    PDSmith said:

    @Zilvergrafix  I did the same method you used, then tried something else and found it works just as well and just as fast.

    I right click on the individual morph folders, and in the parameters section I change the folder to hide, then apply.  all folders and files are now hidden. (how many finger nail morphs do we need? )

    Yup, that's what we've been​ discussing :) It's all a workaround though, I do think something like the old morph injection (like Taoz mentioned) from DAZ's side is the real way to fix this. Or better yet: follow some kind of tree structure to decide when to load morph vertex data or calculate formula relationships. If the "head" of a morph tree is unused, no need to read or calculate any of its children. This is basically what my tool does offline.

    The downside is that the computer probably needs to do some work when you DO want to use a new (potentially complex) morph in your scene. I'd gladly pay that price and I think most people with a large library would.

    Richard Haseltine said:

    That is what is happening now. It's setting up the channels and links that make it work that takes the time.

    I think it's still doing more than it needs to for unused morphs. If not, my simple script wouldn't fix the problem as easily as it does! It also does more than it needs to AFTER loading, like just rotating a limb or loading poses (which is also amplified by having autofollow items). Those problems disappear if the unused morph folders are hidden.

     

    Quick summary of what I found in this thread:

    • Turning of some options (like autofit) can reduce manipulation & animation/pose load times as though all those unused morphs weren't present, though it does not impact scene or figure load times. There's a script uploaded to do/undo that automatically. There's also an alternative workaround using a hidden fit clone which has the same effect (and doesn't need constant fitting/unfitting).

     

    • Hiding morph folders is a non-destructive way to not load them; speeding up scene/figure loads in addition to the above. The 2nd (external) script I referred to analyzes the DUF scene file and automatically hides/unhides morph folders that are unused/used.
    Post edited by grimulkan_9cfbd329bc on
  • IvyIvy Posts: 7,165
    edited February 2021

    Its to bad that Genesis through Genesis 8 series of characters didn't come with a power-loader like generation 4 characters had.

    with the powerloader you can check or uncheck the morphs or characters presets you wanted to load, it saved a ton of load time. and was great for when working on custom characters morphs because you could isolate your morphs from eveyone elses or the preloads. .

    I wonder why Daz stop using the powerloader it was a great feature.

    heres a screen shot of it

    Capture.JPG
    1953 x 1009 - 213K
    Post edited by Ivy on
  • grimulkan_9cfbd329bcgrimulkan_9cfbd329bc Posts: 63

    Mustakettu85 said:

    PowerLoader reborn :)

    Ah! Now I understand this comment. I wonder why DAZ abandoned it too. I mean, their business model is to get us to buy more stuff, right? Even if they make a lot of money from constantly inducting new users with small initial libraries, the big spenders should still matter.

  • Richard HaseltineRichard Haseltine Posts: 100,778

    grimulkan_9cfbd329bc said:

    volpler11 said:

    Did some more poking. 

    The cache file is basically most of the files in "data\DAZ 3D\Genesis 8\Female\Morphs\...." mashed together with one important exception. The bulk of the data in those files are of two types: morph vertex delta and formula. The important exception is that morph vertex deltas are not included in the cache file and is loaded on demand when you change the slider. Formula data still make into the cache and can slow things down significantly. 

    This lead to one important observation: the load time is proportional to the number of files in the morph folder but not necessarily file size. If the file contain large amount of delta that is not included in the cache it will be fast to load. On the other hand if the file contain lots of formulas it will be very slow to load.

    Edit:Also the morph file could be compressed which would make it smaller than it actually are.

    This is gold (as is your thread)! Thanks.

    When I switch between morph folder setups using my tool, I don't really see a lot of time being spent re-creating the cache. Maybe it's because the morphs are still expected to "be there", since they're just hidden, and the cache isn't modified? So maybe DAZ just doesn't bother loading the vertex data or truncates formula calculation if the file is hidden. Or maybe cache creation is only like 20% of the total load time so I didn't notice.

    But your caching explanation explains why my 2nd G8F load in my original test loaded faster than the first. Also, I was parsing the wrong file (log) to try and figure load times, will try again with the cache file. Did someone write a tool for this once?

    PDSmith said:

    @Zilvergrafix  I did the same method you used, then tried something else and found it works just as well and just as fast.

    I right click on the individual morph folders, and in the parameters section I change the folder to hide, then apply.  all folders and files are now hidden. (how many finger nail morphs do we need? )

    Yup, that's what we've been​ discussing :) It's all a workaround though, I do think something like the old morph injection (like Taoz mentioned) from DAZ's side is the real way to fix this. Or better yet: follow some kind of tree structure to decide when to load morph vertex data or calculate formula relationships. If the "head" of a morph tree is unused, no need to read or calculate any of its children. This is basically what my tool does offline.

    The downside is that the computer probably needs to do some work when you DO want to use a new (potentially complex) morph in your scene. I'd gladly pay that price and I think most people with a large library would.

    Richard Haseltine said:

    That is what is happening now. It's setting up the channels and links that make it work that takes the time.

    I think it's still doing more than it needs to for unused morphs. If not, my simple script wouldn't fix the problem as easily as it does! It also does more than it needs to AFTER loading, like just rotating a limb or loading poses (which is also amplified by having autofollow items). Those problems disappear if the unused morph folders are hidden.

     

    Quick summary of what I found in this thread:

    • Turning of some options (like autofit) can reduce manipulation & animation/pose load times as though all those unused morphs weren't present, though it does not impact scene or figure load times. There's a script uploaded to do/undo that automatically. There's also an alternative workaround using a hidden fit clone which has the same effect (and doesn't need constant fitting/unfitting).

     

    • Hiding morph folders is a non-destructive way to not load them; speeding up scene/figure loads in addition to the above. The 2nd (external) script I referred to analyzes the DUF scene file and automatically hides/unhides morph folders that are unused/used.

    If you hide morphs then they aren't read adn so DS doesn't take time setting up the channels and links - your solution works because the time is consumed as I stated, they are not conflicting accounts of the same phenomenon.

  • grimulkan_9cfbd329bcgrimulkan_9cfbd329bc Posts: 63
    edited February 2021

    Richard Haseltine said:

    If you hide morphs then they aren't read adn so DS doesn't take time setting up the channels and links - your solution works because the time is consumed as I stated, they are not conflicting accounts of the same phenomenon.

    Do agree, but the point is they shouldn't be trying to calculate those links in the first place, if unused. Maybe I'm just misunderstanding how morph injection used to work and we're both saying the same thing.

    It's almost like DAZ decided to prioritize pre-calculation of said links, to make turning morphs on/off quicker for the end user. Hence the move away from manual injection & power loader. But as the number of available products and content libraries grew in size over time, the decision began to penalize load times (as well as posing and animation). Maybe only significantly penalizing a small number of users with very large libraries.

    Post edited by grimulkan_9cfbd329bc on
  • TaozTaoz Posts: 9,940
    edited February 2021

    Never mind, answered in a later post...

    Post edited by Taoz on
  • TaozTaoz Posts: 9,940

    grimulkan_9cfbd329bc said:

    Richard Haseltine said:

    If you hide morphs then they aren't read adn so DS doesn't take time setting up the channels and links - your solution works because the time is consumed as I stated, they are not conflicting accounts of the same phenomenon.

    Do agree, but the point is they shouldn't be trying to calculate those links in the first place, if unused. Maybe I'm just misunderstanding how morph injection used to work and we're both saying the same thing.

    As I understand it it's setting up the dials for all the morphs that takes time, and if you don't do that there's no way to manage them in DS.  Whatever you want to do with the morph (apply, inject/remove, hide/unhide) there has to be some kind of reference in DS from where you can manage it, if not a dial (as it is now) then a checkbox or whatever.  But in any case there must be a faster way to handle a large number of morphs. 

  • Richard HaseltineRichard Haseltine Posts: 100,778

    grimulkan_9cfbd329bc said:

    Richard Haseltine said:

    If you hide morphs then they aren't read adn so DS doesn't take time setting up the channels and links - your solution works because the time is consumed as I stated, they are not conflicting accounts of the same phenomenon.

    Do agree, but the point is they shouldn't be trying to calculate those links in the first place, if unused. Maybe I'm just misunderstanding how morph injection used to work and we're both saying the same thing.

    It's almost like DAZ decided to prioritize pre-calculation of said links, to make turning morphs on/off quicker for the end user. Hence the move away from manual injection & power loader. But as the number of available products and content libraries grew in size over time, the decision began to penalize load times (as well as posing and animation). Maybe only significantly penalizing a small number of users with very large libraries.

    The links need to be loaded so that DS knows how the property should affect others and is affected by others. If there is a constant term in the link then the value of the property itself does not have to be non-zero for it to have an effect, so waiting until it is used is not a viable strategy.

  • jbowlerjbowler Posts: 794

    volpler11 said:

    I have done some previous investigation into figure load time at here.

    My main findings are:

    1. most of the time is spent loading a cache file located at "AppData\Roaming\DAZ 3D\dson\cache\DAZ 3D\Genesis 8\"

    Hum... On my system I moved the cache directory to G:/Daz 3D/dson/cache (this is set in Preferences/General tab/DSON Cache Files).  The only thing in that directory is a sub-directory called data; there is no DAZ 3D subdirectory.  Under data there are sub-directories named by PA, including DAZ 3D, however not all the PAs are there and all of the files I checked including the ones under DAZ 3D seem to be simple .dsf files pertaining to wearables (wardrobe mostly but including eyebrows, tears in the DAZ 3D folder.)

    What is the cache file called?  I did turn on "show hidden files".

    I also don't see how a single cache file can explain what I see; I can, and normally do, run two or more instances of DAZStudio at once and those separate instances show timing based on their own Content Directory Manager directory sets.  My default Content Directory Management set of directories is just sufficient for Genesis 8 with the figures I've used before; so I have a lot more G8 figures which I have never used in any scenes that I've kept, butthose are exclude from the default along with most G3 and all G2, G and pre-G characters, wardrobe, props etc.  I ran three loads in parallel loading a test scene with just one Genesis 8 Basic Female but with three different Content Directory Management settings then got the load times from the log files.  The timing is the difference between these two log messages:

    2021-02-07 15:21:31.962 *** Scene Cleared ***
    2021-02-07 15:21:52.924 Loaded image G8FBaseEyelashes_1006.jpg

    That's maybe not quite all the time until I get the UI back after the load but the 21s above is pretty much in agreement with the wall clock times I have got today using a stopwatch; 19s to 25s in that case.

    1. The default settings: just the G8 characters I commonly use.  This is the timings above; 21 seconds.
    2. With the Content Directory Management directory list set to "Everything", all the directories I've got stuff installed to; 127 seconds.
    3. With the Content Directory Management directory list set to "Everything" then changed back to "Default", so it ends up the same as (1); 28 seconds.
    4. As a separate test (not at the same time), using the running DAZStudio instance from (2) I deleted the G8F (to eliminate most of the scene clearing time) and changed back to "Default" (as in test (3)) then loaded the test file; 21 seconds.

    I also took a close look at the AppData runtime directories; there is nothing in those directories that could be a cache of the type discussed.  In particular all the files are copies of files from the main Studio4 runtime directory except for the log files and web_cache/WebpageIcons.db which is at 17408 byte SQLite file with pretty much nothing in it.​

  • grimulkan_9cfbd329bcgrimulkan_9cfbd329bc Posts: 63
    edited February 2021

    Richard Haseltine said:

    The links need to be loaded so that DS knows how the property should affect others and is affected by others. If there is a constant term in the link then the value of the property itself does not have to be non-zero for it to have an effect, so waiting until it is used is not a viable strategy.

    I see. So the only reason my script fixes the problem is because the links are sparse. Note that I don't compute the final value of the morph (which, as you say, took too long). I just check if there is a possible path for a morph to acquire a non-default value from the saved scene data, which only takes me a few seconds even on larger scenes. Morphs which cannot possibly be set to a non-default value via a forumla, assuming there are no scripts outside the scene file that manipulate morphs, are the only ones that are hidden. This can indeed help when loading or posing/animating. 

    The price to be paid is that the sparsity goes down as the user uses more morphs, and some additional lag when this happens due to recalculations (in the case of my script, it will need a restart of Studio when new morphs are to be added, but DAZ themselves don't have that restriction).

    That's all strictly better than the current situation, I don't see why it wouldn't be viable.

    EDIT: Okay, I do build the "sparsity matrix" offline which takes ~5 minutes, which is about how long DAZ takes to load everything. But once I build the matrix, with timestamps for each morph file's modified date, updating the database takes only a few seconds when morphs are added/updated.

     

    TLDR: DAZ could exploit link sparsity (a list of what morphs can possibly affect what other morphs) to optimize their loading & re-calculation process.

    Post edited by grimulkan_9cfbd329bc on
  • ZilvergrafixZilvergrafix Posts: 1,385
    edited February 2021

    PDSmith said:

    @Zilvergrafix  I did the same method you used, then tried something else and found it works just as well and just as fast.

    I right click on the individual morph folders, and in the parameters section I change the folder to hide, then apply.  all folders and files are now hidden. (how many finger nail morphs do we need? )

    Hide morphs is great with 1 or 3 characters, not with +100 I have heavily customized, but I agree with your example of finger nails, for me is like +200 Eyelashes morphs and +150 nipple morphs that are useless due to the abomination of geometry on the nipples at such grade that I needed to buy a geograft from Rotica just for satisfy...satisfy...others cheeky

    let's move on the initial method:

    just beware because I had no access to a beautiful customized character of mine when I saved with hidding morphs and reinstalling got me an error that DS is not able to fix and me too, the character file is broken (prematurely or officialy got corrupted) and no way to create again, I saved minutes in initial load but... I lost a character. crying

     

    Post edited by Zilvergrafix on
  • volpler11volpler11 Posts: 29

    jbowler said:

    volpler11 said:

    I have done some previous investigation into figure load time at here.

    My main findings are:

    1. most of the time is spent loading a cache file located at "AppData\Roaming\DAZ 3D\dson\cache\DAZ 3D\Genesis 8\"

    Hum... On my system I moved the cache directory to G:/Daz 3D/dson/cache (this is set in Preferences/General tab/DSON Cache Files).  The only thing in that directory is a sub-directory called data; there is no DAZ 3D subdirectory.  Under data there are sub-directories named by PA, including DAZ 3D, however not all the PAs are there and all of the files I checked including the ones under DAZ 3D seem to be simple .dsf files pertaining to wearables (wardrobe mostly but including eyebrows, tears in the DAZ 3D folder.)

    What is the cache file called?  I did turn on "show hidden files".

    I also don't see how a single cache file can explain what I see; I can, and normally do, run two or more instances of DAZStudio at once and those separate instances show timing based on their own Content Directory Manager directory sets.  My default Content Directory Management set of directories is just sufficient for Genesis 8 with the figures I've used before; so I have a lot more G8 figures which I have never used in any scenes that I've kept, butthose are exclude from the default along with most G3 and all G2, G and pre-G characters, wardrobe, props etc.  I ran three loads in parallel loading a test scene with just one Genesis 8 Basic Female but with three different Content Directory Management settings then got the load times from the log files.  The timing is the difference between these two log messages:

    2021-02-07 15:21:31.962 *** Scene Cleared ***
    2021-02-07 15:21:52.924 Loaded image G8FBaseEyelashes_1006.jpg

    That's maybe not quite all the time until I get the UI back after the load but the 21s above is pretty much in agreement with the wall clock times I have got today using a stopwatch; 19s to 25s in that case.

    1. The default settings: just the G8 characters I commonly use.  This is the timings above; 21 seconds.
    2. With the Content Directory Management directory list set to "Everything", all the directories I've got stuff installed to; 127 seconds.
    3. With the Content Directory Management directory list set to "Everything" then changed back to "Default", so it ends up the same as (1); 28 seconds.
    4. As a separate test (not at the same time), using the running DAZStudio instance from (2) I deleted the G8F (to eliminate most of the scene clearing time) and changed back to "Default" (as in test (3)) then loaded the test file; 21 seconds.

    I also took a close look at the AppData runtime directories; there is nothing in those dire+ctories that could be a cache of the type discussed.  In particular all the files are copies of files from the main Studio4 runtime directory except for the log files and web_cache/WebpageIcons.db which is at 17408 byte SQLite file with pretty much nothing in it.​

    The file is called "Female.cache.dsf"

  • volpler11volpler11 Posts: 29
    edited February 2021

    Zilvergrafix said:

    PDSmith said:

    @Zilvergrafix  I did the same method you used, then tried something else and found it works just as well and just as fast.

    I right click on the individual morph folders, and in the parameters section I change the folder to hide, then apply.  all folders and files are now hidden. (how many finger nail morphs do we need? )

    Hide morphs is great with 1 or 3 characters, not with +100 I have heavily customized, but I agree with your example of finger nails, for me is like +200 Eyelashes morphs and +150 nipple morphs that are useless due to the abomination of geometry on the nipples at such grade that I needed to buy a geograft from Rotica just for satisfy...satisfy...others cheeky

    let's move on the initial method:

    just beware because I had no access to a beautiful customized character of mine when I saved with hidding morphs and reinstalling got me an error that DS is not able to fix and me too, the character file is broken (prematurely or officialy got corrupted) and no way to create again, I saved minutes in initial load but... I lost a character. crying

    I think i come up with a way to use characters without performance impact with very little incovenience. It basically involves making copy of the genesis 8 figure. I only saw someone suggesting doing this manually once when I was originally researching this problem but I can't find it now.

    Morphs on the other hand are a bit more difficult to deal with. I did make a python script to analyse the cache file to find which product is causing the most of the slow down. Posted in the my other thread.

    Post edited by volpler11 on
  • jbowlerjbowler Posts: 794

    Richard Haseltine said:

    grimulkan_9cfbd329bc said:

    Richard Haseltine said:

    If you hide morphs then they aren't read adn so DS doesn't take time setting up the channels and links - your solution works because the time is consumed as I stated, they are not conflicting accounts of the same phenomenon.

    Do agree, but the point is they shouldn't be trying to calculate those links in the first place, if unused. Maybe I'm just misunderstanding how morph injection used to work and we're both saying the same thing.

    It's almost like DAZ decided to prioritize pre-calculation of said links, to make turning morphs on/off quicker for the end user. Hence the move away from manual injection & power loader. But as the number of available products and content libraries grew in size over time, the decision began to penalize load times (as well as posing and animation). Maybe only significantly penalizing a small number of users with very large libraries.

    The links need to be loaded so that DS knows how the property should affect others and is affected by others. If there is a constant term in the link then the value of the property itself does not have to be non-zero for it to have an effect, so waiting until it is used is not a viable strategy.

    This is a software engineering analysis and I won't dispute its correctness.

    However my experiments show that I get a substantial speed-up on both scene load and character insert by excluding just character morph controls; I'm not excluding all character morphs and I have a set of "generic" morphs such as the Musculature morphs loaded.  Obviously the results I get depend on my particular CMS; what I've bought.  I do know from previous experiments that I can get character insertion time down to less than 10s by excluding pretty much everything but that would be pointless.

    So I take off my "computer programmer" hat and put on my "user" hat.  It's a nicer hat because I'm always right when I have that one on.

    With that hat on the issue is simply that I don't want to see my load times increase simply because I bought a new character.  Sure, if I buy a morph set that I can apply to any character and any scene including ones I have already composed, rendered and am satisfied with, sure, then I will take a small hit in load time; I bought the morph set for that purpose.  But a new character is a completely different thing.  I don't expect buying Hugren to slow down the load time of every G8M character I have and, worse, I expect Hugren to stay in his corner and look pretty; I'm perfectly happy if he doesn't suddently appear as a morph option on every G8M in the scene (which  is what happens at present).

    I'm perfectly happy with THIS behavior:

    imageimage

    Those two screen shots are from exactly the same instance of DAZStudio, all I did for the second was to select male (2)...  I inserted male 1 with my default set of male characters I then just changed the CMS to include Everything I have purchased and inserted male 2.  Ok.  That's fine.  But if I save that file I believe both G8Ms will have all the options (I didn't test that).

    What I want is for the behavior above to persist.  I don't want options automagically added to scenes I have already made AND I don't want character options to be stored in a scene unless I set them.  In fact it's been a long day but I think DAZStudio 4.15 does not do the latter, however either earlier versions did or I've yet to discover something (dForcing maybe?) that forces those options in.

    G8M with default CMS.JPG
    1654 x 1400 - 230K
    G8M with Everything CMS.JPG
    1654 x 1400 - 256K
  • volpler11volpler11 Posts: 29
    edited February 2021

    How to load character fast without deleting characters [WIP]:

    I will provide some background explaination and then propose a workaround. I will use genesis 8 female as an example.

    The issue:

    The more morphs a figure have, the longer it will take to load. That is some content in "data\DAZ 3D\Genesis 8\Female\Morphs\" contributes to the load time. 

    The content of files in the morphs folder can be seperated 3 categories: morph deltas, formulas, and metadata\others. Daz will cache the morphs in a file called "Female.cache.dsf" in "AppData\Roaming\DAZ 3D\dson\cache\data\DAZ 3D\Genesis 8". It basically combines the matadata and formulas of all the files in the morph folder into one file. Morph deltas are not included. Daz will parse this cache file and this is unfortunately a single thread slow process. The cache file contain the last modified time of a file so Daz will know which file need to be updated. If there are new morphs Daz will load them and write out a new cache file.

    Morph deltas looks like this and is loaded when you change a slider so does not appear to impact load time.

    "morph" : {
    "vertex_count" : -1,
    "deltas" : {
    "count" : 11016,
    "values" : [
        [ 0, 0.0003357831, -1.385933, 0.9539337 ],
        [ 1, -0.0009604891, -1.793137, 1.383705 ],
        [ 4, 0.0002822045, -2.334381, -0.2555213 ],
... skip thousands of lines ...

    Formulas looks like this and is a main contributor to load time. Some character or morph sets can have handreds of formulas.

    "formulas" : [
{
    "output" : "hip:/data/DAZ%203D/Genesis%208/Female/Genesis8Female.dsf#hip?center_point/y",
    "operations" : [
        { "op" : "push", "url" : "Genesis8Female:#FBMSue?value" },
        { "op" : "push", "val" : -1.337715 },
        { "op" : "mult" }
]
},

    and metadata looks like this:

    "assets" : [
{
    "file_version" : "0.6.0.0",
    "asset_info" : {
        "id" : "/data/DAZ%203D/Genesis%208/Female/Morphs/DAZ%203D/Base%20Correctives/lShldrBend_CTRLMD_N_YRotate_n110.dsf",
        "type" : "modifier",
        "contributor" : {
            "author" : "Daz 3D",
            "email" : "",
            "website" : "www.daz3d.com"
        },
        "revision" : "1.0",
        "modified" : "2021-02-07T12:27:36Z"
    },
    "modifier_library" : [
    {
        "id" : "CTRLMD_N_YRotate_n110",
        "name" : "CTRLMD_N_YRotate_n110",
        "parent" : "/data/DAZ%203D/Genesis%208/Female/Genesis8Female.dsf#lShldr",
        "presentation" : {
            "type" : "Modifier/Corrective",
            "label" : "",
            "description" : "",
            "icon_large" : "",
            "colors" : [ [ 0.5019608, 0, 0 ], [ 0.5019608, 0, 0 ] ]
        },
        "channel" : {
            "id" : "value",
            "type" : "float",
            "name" : "CTRLMD_N_YRotate_n110",
            "label" : "CTRLMD_N_YRotate_n110",
            "visible" : false,
            "locked" : true,
            "value" : 0,
            "clamped" : true,
            "min" : 0,
            "max" : 1,
            "step_size" : 0.01
        },
        "group" : "/Hidden/CTRLMDs",
        "formulas" : [
... skipped ....
        }
    ],
    "scene" : {
        "modifiers" : [
        {
            "id" : "CTRLMD_N_YRotate_n110-1",
            "url" : "#CTRLMD_N_YRotate_n110"
}
]
}
},

    There are two main types of product that add morphs: characters and morph sets. Both can add a large number of formula to the figure causing load time bloat.

    Post edited by volpler11 on
  • jbowlerjbowler Posts: 794

    volpler11 said:

    The file is called "Female.cache.dsf"

    Found it:

    G:\DAZ 3D\dson\cache\data\DAZ 3D\Genesis 8\Female.cache.dsf, it's 104,625,417 bytes with my setup, curiously in the same directory Female 8_1.cache.dsf is 346,380,801 bytes and was last updated two days ago.

    It gets created when a G8F is inserted into the scene; I tested this by deleting it; it came back at the same size.  So I swapped to my "Everything" set of content directories, deleted it again and inserted Genesis 8 Basic Female; it came back at 350,539,905 bytes, but it took a lot longer.

    So I messed some more with my Contect Directory Manager directories to eliminate pretty much everything I didn't need to insert the Genesis 8 Basic Female (it's just a few checkbox clicks for me now...) and inserted a G8 Basic Female, boy was that fast, and the cache file came back at 23,344,247 bytes.

    I don't think the contents of the cache file are changing that much; what matters is the CMS directories.  I saved the above 3xG8F scene, closed DAZStudio and reopened it the reloaded the scene, that took 40s (wall clock time) and the cache file was regenerated back to the 104,625,417 byte size (i.e. the size corresponding to my "default" CMS directory setting).  So the cache file is updated if DAZStudio finds it was missing stuff from the CMS, therefore it will gradually get bigger on my system as I load or create scenes with the "Everything" setting, but the time involved doesn't seem to be great.  In fact the cache doesn't seem to be making a whole lot of difference to the times!

    Given that I now understand what it does I'll probably move it off my content SSD (which is fast) to a temporary directory on my C: SSD (which is very fast) and make sure I don't accidentally back it up.

  • volpler11volpler11 Posts: 29
    edited March 2021

    ---

    f1.PNG
    683 x 809 - 67K
    f2.PNG
    316 x 472 - 20K
    Post edited by volpler11 on
  • volpler11volpler11 Posts: 29

    jbowler said:

    it came back at 350,539,905 bytes, but it took a lot longer.

    boy was that fast, and the cache file came back at 23,344,247 bytes.

    Large amount of morph instead lead to large cache file which is slower to load. Less morphs installed, smaller cache, fast load time. A fresh copy of genesis 8 female is only 3mb in cache file size. The cache file is basicly a copy of all the files in the morph folder so daz dont have to open thousands of small file, it will just compare the file modification time to do a quick check at the end and load files not in the cache and rebuild the cache file. Keep in mind building the cache file is slow, its better to do performance measure after the cache file have been rebuilt.

    I think its parsing the cache file that is slow, so storage speed might not help, single core cpu performance is the bottleneck.

    My theory is by looking at the content of the cache file, it is possible to estimate how much impact a particular item have on load time. I am assuming more room it take, the longer it will take to load.

  • TaozTaoz Posts: 9,940

    volpler11 said:

     Daz will parse this cache file and this is unfortunately a single thread slow process.

    Multithreading would probably make a significant difference here and shouldn't be too difficult to implement in this case, I think.

  • Richard HaseltineRichard Haseltine Posts: 100,778

    Taoz said:

    volpler11 said:

     Daz will parse this cache file and this is unfortunately a single thread slow process.

    Multithreading would probably make a significant difference here and shouldn't be too difficult to implement in this case, I think.

    I'm not sure, keeping the threads from overlapping as they traced through dependencies sounds - at best - difficult.

  • jbowlerjbowler Posts: 794

    Richard Haseltine said:

    Taoz said:

    volpler11 said:

     Daz will parse this cache file and this is unfortunately a single thread slow process.

    Multithreading would probably make a significant difference here and shouldn't be too difficult to implement in this case, I think.

    I'm not sure, keeping the threads from overlapping as they traced through dependencies sounds - at best - difficult.

    They can overlap; that's just a matter of updating a data structure from multiple threads and that is pretty easy with Qt.  As @grimulkan_9cfbd329bc observed:

     

    grimulkan_9cfbd329bc said:

    Richard Haseltine said:

    The links need to be loaded so that DS knows how the property should affect others and is affected by others. If there is a constant term in the link then the value of the property itself does not have to be non-zero for it to have an effect, so waiting until it is used is not a viable strategy.

    I see. So the only reason my script fixes the problem is because the links are sparse. Note that I don't compute the final value of the morph (which, as you say, took too long). I just check if there is a possible path for a morph to acquire a non-default value from the saved scene data, which only takes me a few seconds even on larger scenes. Morphs which cannot possibly be set to a non-default value via a forumla, assuming there are no scripts outside the scene file that manipulate morphs, are the only ones that are hidden. This can indeed help when loading or posing/animating. 

    And, indeed, it looks like the vast majority of the time goes handling the formulae; I tried a different, "complex scene" test (five G8 characters) without the cache and it took 96 seconds before the first "Could not find parent for forumla" message appeared.  The entire cache was built in that time (from the times of the cache.dsf files on disk).  It then took 286 seconds outputing those messages until it started loading the first morph delta description (off disk).  Loading from that point to the end of the load took a final 115 seconds.

    Given that most of the slow-down comes from things that won't affect the scene and that there are a significant number of log messages pretty much saying that the formula is a no-op (it has no place to go in the scene) it seems like an ideal candidate for multi-threading.  Of course that is still a software-engineer solution; the doctor's solution, "don't do that", is probably even easier...

    Here are the timings with and without the DSON cache files; I ran two sets of tests once with a cleared cache (so that built the complete cache on disk) then I restarted DAZStudio and loaded the scene again, so that should have used the now populated cache.  I verified this by checking the data stamps on the .cache.dsf files; they had not been rewritten.

    NO and YES columns contain the absolute time (day, time of day) from the DAZStudio log.  "NO" contains entries with no cache, "YES" contains entries from a subsequent run when the cache was populated.  The Delta colum contains the time difference in seconds between successive rows except that the last row is the total load time in minutes:seconds.  The TAG column records the identifying message from the log file; the time of the first entry with that message is what is recorded in the NO and YES columns.
    NO Delta(no)s YES Delta(yes)s Delta change (s) TAG
    52:07.676   05:20.438     *** Scene Cleared ***
    52:13.094 5.418 05:25.596 5.158 -0.260 JSON interpreter warning: Unexpected string member: type
    53:26.312 73.218 05:56.486 30.890 -42.328 dzassetdaz.cpp(5836): Could not find parent for modifier
    53:43.546 17.234 06:13.770 17.284 0.050 dzassetdaz.cpp(6777): Could not find output property for formula
    58:29.289 285.743 11:09.625 295.855 10.112 Loaded Morph Deltas in 0 min 0.0 sec.
    59:11.188 41.899 11:51.934 42.309 0.410 Loaded file: FBM Mika 8 Body.dsf
    00:24.360 73.172 12:39.936 48.002 -25.170 Loaded image SSH BLOND 1.jpg
      08:16.684   07:19.498 -57.186 TOTAL SCENE LOAD TIME (mm:ss.000)

    The DSON cache does make a difference to the "find parents for modifiers" step - down from 73s when the cache to to be built to 31s with it present.  Based on other experiments that difference does disappear if (just) the DAZ 3D/Genesis cache files have to be rebuilt and this happens when I change content directory manager settings.  The last line, time to load the scene, almost certainly just reflects the fact that I had populated the operating system file cache with the files that needed to be loaded.  It can be seen that the big number in the whole load is the time to resolve the formulae; 290 seconds, two thirds of the total load time.

     

  • RobotHeadArtRobotHeadArt Posts: 917
    edited February 2021

    Here is one way DAZ could redesign the morph loading.

    When double clicking a Genesis figure (since it wouldn't necessarily be limited to just G8/G8.1) a dialog box will display. A checkbox would say "Load Morphs Asynchronously".  If unchecked, the figure loads same behavior as today. If checked, the figure and the base morphs that ship with the figure loads in the viewport. A background thread would then begin the drudgery of loading the parsing the morph files and doing the morph formula linking.  The parameters tab for the Genesis figure would show an animated spinner as the morphs load and the non-base figure morphs would be unavailble for use until the loading process is complete.  This would allow the end user to begin applying hair, clothing, poses, positioning the figure in the scene, apply materials, test rendering, etc. while the morphs load in the background.

    Bonus round: when DAZ Studio encounters a bad formula (like a link to a product you don't have installed) it gets put on a do not load list. The do not load list morphs would not be parsed the next time the figure loads.  This would greatly speed up morph loading in general and reduce the bad formula spam to the error log.  There could be another checkbox on the Genesis load dialog box that would say "Load Skipped Morphs" which would control this behavior (so a user who later installs a linked morph can then pick it up or if a vendor supplied a bugfix that corrected a broken morph link).

    Post edited by RobotHeadArt on
  • PerttiAPerttiA Posts: 10,024
    edited February 2021

    RobotHeadArt said:

    Bonus round: when DAZ Studio encounters a bad formula (like a link to a product you don't have installed) it gets put on a do not load list. The do not load list morphs would not be parsed the next time the figure loads.  This would greatly speed up morph loading in general and reduce the bad formula spam to the error log.  There could be another checkbox on the Genesis load dialog box that would say "Load Skipped Morphs" which would control this behavior (so a user who later installs a linked morph can then pick it up or if a vendor supplied a bugfix that corrected a broken morph link).

    Which morph would that be, the one calling the product you don't have or the one you don't have - Putting the one you don't have on a do not load list would be pointless, and the first one if put on a list, would in effect remove the morph (Say, Katie for Topsy 8 if you don't have Topsy).

    Post edited by PerttiA on
  • jbowlerjbowler Posts: 794

    PerttiA said:

    RobotHeadArt said:

    Bonus round: when DAZ Studio encounters a bad formula (like a link to a product you don't have installed) it gets put on a do not load list. The do not load list morphs would not be parsed the next time the figure loads.  This would greatly speed up morph loading in general and reduce the bad formula spam to the error log.  There could be another checkbox on the Genesis load dialog box that would say "Load Skipped Morphs" which would control this behavior (so a user who later installs a linked morph can then pick it up or if a vendor supplied a bugfix that corrected a broken morph link).

    Which morph would that be, the one calling the product you don't have or the one you don't have - Putting the one you don't have on a do not load list would be pointless, and the first one if put on a list, would in effect remove the morph (Say, Katie for Topsy 8 if you don't have Topsy).

    I don't think bad formulae are the issue; I do get a lot of those warnings but some of them might simply be because the "Upper Face Rig" on my G8 characters keeps disappearing from the scene view and the parameters pane (the nodes are still there in the timeline).  In the scene in question there are five G8 characters but one of them doesn't  have a visible "Upper Face Rig" bone.  Regardless of that I only get 1038 "Could not find target property for formula" warnings and that hardly seems enough to cause any significant delay.

    What seems to be happening is that DAZStudio has built an in-memory cache of the "modifier" assets and all the "forumla" assets then runs through that enormous data structure building the control structure for each inserted character.  So the on-disk cache helps shave a minute or so off the initial in-memory cache build but the per-character scanning of that tree takes all the time.

    Apart from multi-threading this there are probably other approaches along the lines of delay loading formulae until a modifier is actually changed.  That would be almost as good as not loading information for character morphs until the morph was actually used; something like a check-box to switch on the morphs for a character.

     

  • ZilvergrafixZilvergrafix Posts: 1,385
    edited February 2021

    I've fixed the problem just making a backup of my custom characters and loading one by one, probably this method only works for me and my workflow because I use the same morphs over and over, so, when I load a second custom character  there are already shared morphs when I backed up the first one. Yes, normal characters not from Daz (because I leave intact that one only) but other stores won't load until I allow rellocate the morphs that they need to make them work.

     

     

    Post edited by Zilvergrafix on
  • volpler11volpler11 Posts: 29

    I think bad formula is a red herring. Sometime a bad formula is the result of formula intended for a character you have not installed, but installing the character could actually make the load time even longer.

    The main problem is what jbowler have discovered that there are just too much formula to load. If you look at the cache file content, bulk of it is formulas. 

    @jbowler can you have a look at the log produced by procmon and see what insight you come up with.

  • jbowlerjbowler Posts: 794

    volpler11 said:

    I think bad formula is a red herring. Sometime a bad formula is the result of formula intended for a character you have not installed, but installing the character could actually make the load time even longer.

    The main problem is what jbowler have discovered that there are just too much formula to load. If you look at the cache file content, bulk of it is formulas. 

    @jbowler can you have a look at the log produced by procmon and see what insight you come up with.

    It's pretty much what I would expect.  I did run a different version of my test file but it has all the same content (I'm not sure what was changed between the two; they are only a few minutes apart).  Pretty much:

    10:04:30  DAZStudio starts to open the test file
    10:04:32  Starts to open .dsf files, queries but does not read texture files.  It looks like DAZStudio is doing a quick check for missing files at this point.
    10:04:33 Finishes reading .dsf files for the scene and starts on the first (G8M) character; this is just a feature of the order of things in the scene.
    10:06:05 Starts writing Female.cache.dsf
    10:06:23 Ends writing Female.cache.dsf and continues reading more .dsf files and periodically writing .cache.dsf files
    10:06:25 Sends some stuff to googleusercontent.com (i.e. on the Internet) then starts looking for textures again.  Some of this includes looking in the DAZ Connecct library, which can't be removed but which I have disabled (so the searches get PATH NOT FOUND).  For some reason it also tries looking in C:\ - not sure where that comes from...  It also looks for scripts (.dsa, .dse) during this process.
    10:06:34 Finished looking for things, writes to the log file for a while.  Apart from that the only recording thing is thread creation, around 38 new threads (so maybe it is multi-threading the formula handling).
    10:11:50 End of period of silence... It looks like it is setting up for rendering (base on the registery entries being read).  It tests for .dsf and .dhdm files, most tests fail - it tries to locate each file in each of my content management directories (so 43 tests to find out the file isn't there).
    10:13:11 Starts actually reading texture files.
    10:14:01 Finishes reading SSH BLOND 1.jpg; the last file in my log messages.
    10:14:33 This seems to be the end of the load

    So the end of the cache read is probably the message to googleusercontent.com almost 2 minutes after the start and the formula handling is a purely CPU activity starting then which lasts 5.5 minutes.  The next step, reading the .dsf files, takes 1 minute 20s and finally there is another 50 seconds to load the textures.  I can't be completely sure about all of this; there are almost 1 million lines in the log.  The multiple threads created at the start of the formula handling most likely reflects multi-threading for some other activity because the actual CPU usage on my system sticks at about 11% during the formula activity; I think that corresponds to one CPU running flat out.  CPU activity does go much higher after the formula activity; it looks like it peaks at around 40%.

     

  • volpler11volpler11 Posts: 29

    Its me again.

    I just had an idea of making a plugin that optionally load morph based on the method of making a new figure. It should be doable.

    Just need to:

    1.Duplicate the base figure as a new figure (at least doable manually).

    2. Figure out how to find all the morphs for the figure. (dont know how)

    3. Let user select which morph to add to the character (should be doable but dont know how).

    4. Apply the morphs to the character (doable with dazscript).

     

  • fablemanfableman Posts: 23

    Daze really need to change something, this is just getting worse and worse for everyone.

Sign In or Register to comment.