Duplicate Formulas - possible solution

TaozTaoz Posts: 9,979
edited May 2021 in The Commons

A possible solution to the Duplicate Formula problem:

Extract each product file, scan through all data, everywhere there is a reference that can cause the Duplicate Formula you append an ID of say 4 random characters to that reference, as well as to what it refers to, so they match.  E.g. if we take a filename as example the file name and it's reference(s) would be changed from "ThisCharacter.png" to e.g. "ThisCharacter(d4ls).png".  Then rewrite the file with the modified data.  There may be metadata that need to be changed also but that shouldn't be a big problem.

The random ID could be the same for all references in a product (a unique product ID), or unique for each reference. The risk that there are two products with both the same file name and the same random ID is so small that it can be ignored, and even if it should happen you just change the ID(s) for one of the products.

Would this work?

 

Post edited by Taoz on

Comments

  • PerttiAPerttiA Posts: 10,024

    The file name doesn't matter.

  • TaozTaoz Posts: 9,979

    PerttiA said:

    The file name doesn't matter.

    I was also just demonstrating the principle - if the conflicting entries in question each get a unique ID appended to their name/label they won't clash (which is the problem I assume).

  • vwranglervwrangler Posts: 4,902

    The issue is that the duplicate formulas are massively difficult to find -- I still don't understand why Studio has been designed so that it only tells you the second instance of the duplicate formula occurrence, and not both -- and are references inside a file to different functions. This old thread may illustrate some of the difficulties in figuring out exactly what's going on: https://www.daz3d.com/forums/discussion/comment/2600821/

     

  • TaozTaoz Posts: 9,979

    vwrangler said:

    The issue is that the duplicate formulas are massively difficult to find -- I still don't understand why Studio has been designed so that it only tells you the second instance of the duplicate formula occurrence, and not both -- and are references inside a file to different functions. This old thread may illustrate some of the difficulties in figuring out exactly what's going on: https://www.daz3d.com/forums/discussion/comment/2600821/

     As far as I can read there you get a popup message if you encounter the problem, or what?

  • TaozTaoz Posts: 9,979

    vwrangler said:

    The issue is that the duplicate formulas are massively difficult to find -- I still don't understand why Studio has been designed so that it only tells you the second instance of the duplicate formula occurrence, and not both -- and are references inside a file to different functions. This old thread may illustrate some of the difficulties in figuring out exactly what's going on: https://www.daz3d.com/forums/discussion/comment/2600821/

    Besides, my point was that if you append unique IDs to all entries that can possible clash, in all products, which I believe can be done by some script, the problem should disappear, if name clashing is the cause.

  • Richard HaseltineRichard Haseltine Posts: 102,730

    I don't see how that would cope with products that call other products (and that is a common situation with the Daz base characters and with things like the standard expression packs) - making each product unique would break needed links, assuming each external reference was to a single property with the same name would reintroduce the duplicate IDs conflict in a new form.

  • vwranglervwrangler Posts: 4,902

    Taoz said:

    vwrangler said:

    The issue is that the duplicate formulas are massively difficult to find -- I still don't understand why Studio has been designed so that it only tells you the second instance of the duplicate formula occurrence, and not both -- and are references inside a file to different functions. This old thread may illustrate some of the difficulties in figuring out exactly what's going on: https://www.daz3d.com/forums/discussion/comment/2600821/

     As far as I can read there you get a popup message if you encounter the problem, or what?

     Besides, my point was that if you append unique IDs to all entries that can possible clash, in all products, which I believe can be done by some script, the problem should disappear, if name clashing is the cause.

    Yes, you get a popup if you encounter the issue. It's annoying if you are just doing still renders, but a complete show-stopper -- quite literally -- if you're doing animation, because you have to dismiss the message every time it occurs before you can render the next frame.

    I think one major difficulty with your solution could be that it would need to be done inside the files to every entry that can clash -- and most of the clashes are with common, frequently used functions and formulas. It could lead to some massive file bloat, because you would need to do every function every time, and thus would need to be done many many times inside each DSF used by a given character. Since the most notable occurrences of this are with characters, you'd wind up slowly but surely making load times much higher as you installed more characters. With Genesis 8/8.1, that's already a pretty huge problem.

  • TaozTaoz Posts: 9,979

    Richard Haseltine said:

    I don't see how that would cope with products that call other products (and that is a common situation with the Daz base characters and with things like the standard expression packs) - making each product unique would break needed links, assuming each external reference was to a single property with the same name would reintroduce the duplicate IDs conflict in a new form.

    Products that call other products also do it through references, if these are correctly modified via global search and replace so no links are broken there's no problem.  It may be complex, but I'm sure it can be done one way or another, depending on how the system is designed. 

     

  • TaozTaoz Posts: 9,979

    vwrangler said:

    Taoz said:

    vwrangler said:

    The issue is that the duplicate formulas are massively difficult to find -- I still don't understand why Studio has been designed so that it only tells you the second instance of the duplicate formula occurrence, and not both -- and are references inside a file to different functions. This old thread may illustrate some of the difficulties in figuring out exactly what's going on: https://www.daz3d.com/forums/discussion/comment/2600821/

     As far as I can read there you get a popup message if you encounter the problem, or what?

     Besides, my point was that if you append unique IDs to all entries that can possible clash, in all products, which I believe can be done by some script, the problem should disappear, if name clashing is the cause.

    Yes, you get a popup if you encounter the issue. It's annoying if you are just doing still renders, but a complete show-stopper -- quite literally -- if you're doing animation, because you have to dismiss the message every time it occurs before you can render the next frame.

    OK - that's strange - I have tons of products and characters installed but have never run into such a pop-up.  I always use DIM for installation, never connect.  Couild that be the reason?

    I think one major difficulty with your solution could be that it would need to be done inside the files to every entry that can clash -- and most of the clashes are with common, frequently used functions and formulas. It could lead to some massive file bloat, because you would need to do every function every time, and thus would need to be done many many times inside each DSF used by a given character. Since the most notable occurrences of this are with characters, you'd wind up slowly but surely making load times much higher as you installed more characters. With Genesis 8/8.1, that's already a pretty huge problem.

    I'm assuming clashing in this case is between names or labels that are supposed to be unique and (at least partially) arbitrary but for some reason aren't.  Since the names already are supposed to be unique a rename of a unique name to another unique name that doesn't clash with any another unique name won't make any difference as long as all occurences of the name that is being renamed and all references to it are being updated accordingly.  It would be the same as if you had chosen the name you want to rename it with instead, from the beginning. 

  • Richard HaseltineRichard Haseltine Posts: 102,730

    Taoz said:

    Richard Haseltine said:

    I don't see how that would cope with products that call other products (and that is a common situation with the Daz base characters and with things like the standard expression packs) - making each product unique would break needed links, assuming each external reference was to a single property with the same name would reintroduce the duplicate IDs conflict in a new form.

    Products that call other products also do it through references, if these are correctly modified via global search and replace so no links are broken there's no problem.  It may be complex, but I'm sure it can be done one way or another, depending on how the system is designed. 

    How will it know which of the duplicates?

  • ArtiniArtini Posts: 9,722

    I wonder, if this duplicate formula found problem is only with the message itself.

    One just need to click the pop up to close it. I have rendered many times with such items

    and could not see anything strange on the render.

    Does it means, that such message is just a warning, but has no effect on the render itself?

  • SevrinSevrin Posts: 6,310

    Artini said:

    I wonder, if this duplicate formula found problem is only with the message itself.

    One just need to click the pop up to close it. I have rendered many times with such items

    and could not see anything strange on the render.

    Does it means, that such message is just a warning, but has no effect on the render itself?

    If you're okay with what you see in the viewport, then the render will be fine.  The products aren't functioning 100% correctly, but this type of error doesn't always cause serious functionality problems.

  • ArtiniArtini Posts: 9,722

    Great. I got this problem recently with https://www.daz3d.com/jacqueline-81 and I do not want to uninstall her.

    It will be great to have the possibility to suppress of showing up warnings in Daz Studio in the settings, though.

     

  • PerttiAPerttiA Posts: 10,024

    Artini said:

    Great. I got this problem recently with https://www.daz3d.com/jacqueline-81 and I do not want to uninstall her.

    It will be great to have the possibility to suppress of showing up warnings in Daz Studio in the settings, though.

     The duplicate formulas warning is there for a reason, although having it stop the loading process is not ideal, especially when it doesn't give you any useful information and the only option is to click "ok".

    The warning means that some of the products you have bought and installed are not working as intended due to conflicts with other products you have installed.

  • TaozTaoz Posts: 9,979
    edited May 2021

    Richard Haseltine said:

    Taoz said:

    Richard Haseltine said:

    I don't see how that would cope with products that call other products (and that is a common situation with the Daz base characters and with things like the standard expression packs) - making each product unique would break needed links, assuming each external reference was to a single property with the same name would reintroduce the duplicate IDs conflict in a new form.

    Products that call other products also do it through references, if these are correctly modified via global search and replace so no links are broken there's no problem.  It may be complex, but I'm sure it can be done one way or another, depending on how the system is designed. 

    How will it know which of the duplicates?

    I'm not sure what you mean.  If it's all about clashing names/labels between products you just use names/labels in the product files that are so unique that the risk of a clash between two products is practically zero.  Currently it's easy for PAs to create clashing names, we see it regularly, but it you e.g. append the product SKU, which is unique to every product, to every relevant name/label in a product and whereever these are referenced externally, that product's names/labels will be unique and therefore can't clash with another product's names/labels. 

    If the problem is some kind of duplicate name or reference within the product itself it's a different story of course but then it's the PAs or DAZ' responsibility to check for such things before releasing a product. 

    Post edited by Taoz on
  • Jason GalterioJason Galterio Posts: 2,562

    These errors / warnings have been screwing up a lot of my projects recently so I took the weekend to try and isolate the issues.

    The main takeaway here is this:
    G8F with no errors / warnings loads for me in 2 minutes 45 seconds.
    G8F with errors / messages loads for me in 10 minutes 10 seconds.

    So these errors / warnings are hitting the loading overhead in a significant fashion.

    To put this in perspective, I have ~3,500 Genesis 8 products, all of which I installed to an isolated product directory. I loaded G8F in then pulled up the error log and uninstalled all of the products referenced by Errors and Warnings. I then restarted DS and added a Basic G8F to get the clean 2:45 load time.

    I then reinstalled all of the removed products, about 56 products, to an isolated directory. I then started DS again and loaded a Basic G8F and got the 10 minute load time.

    I posted a full write up on my DA journal if you'd like to see the details, but its a work in progress.

  • Jason GalterioJason Galterio Posts: 2,562
    edited May 2021

    Further experimentation has lead me to a retraction, a correction, and a couple of additional tid bits.

    The highpoints first:

    1. The G8F load time reported in the error log takes into consideration the human interaction time required to click the "Ok" button on the Duplicate Formula error. In other words, the load time clock keeps ticking while waiting for the error to be cleared. In hindsight this is obvious, but just didn't occur to me at the time. The DF errors still have an impact on load times, just not as dramatic as I first thought they were.

    2. DS seems to have a variance of load times on G8F from anywhere between 30 seconds to a minute. I haven't been able to determine what might be causing this, but I wanted to mention it. I usually do multiple loads just to make sure I am not seeing one of these blips.

    3. Having two video cards seems to have an impact on G8F load times. Which isn't something I would have expected.

    4. Missing Target warnings has an impact on the load times, but usually not as dramatic as the Duplicate Formula errors. I say usually only because there are a few products with a lot of these warnings that do have a significant load time impact.

    With that out of the way, here is the new data I found. LT stands for Load Time.

    5 minutes 48 seconds LT: 2 Video cards, Duplicate Formula errors present (and babysat).
    4 minutes 48 seconds LT: 1 Video card, Duplicate Formula errors present (and babysat).
    2 minutes 49 seconds LT: 2 Video cards, no Duplicate Formula errors.
    3 minutes 25 seconds LT: 1 Video card, no Duplicate Formula errors.

    It's interesting that having two video cards makes LT worse with Duplicate Formula errors and better without. Babysat means that I waited for the error message to appear and clicked it as fast as I could. Removing as much of the human interaction time as possible.

    A couple of comments about my hardware / software set up. I have an external secondary video card that I can turn on and off. So the system and software is exactly the same when the video cards are switched. I have somewhere between 650 and 750 G8F character / morph packages installed. Which should explain why the LT would be this long at all.

    For the Duplicate Formula errors, I have three products that have these issues. Two of them have conflict with one other product. The other one I haven't been able to determine the conflict. I'm not going to name the products as it is irrelevent because you have to have both conflicts installed to experience the error.

    Post edited by Jason Galterio on
  • PerttiAPerttiA Posts: 10,024

    It should be noted that if you make changes to your morph folders, either by adding, removing or editing the morph files, the next time you load the figure, the load time is longer because DS updates the useless cache file.

  • Jason GalterioJason Galterio Posts: 2,562

    PerttiA said:

    It should be noted that if you make changes to your morph folders, either by adding, removing or editing the morph files, the next time you load the figure, the load time is longer because DS updates the useless cache file.

    That might explain the random deviations. I was thinking that DS might be busy importing metadata or updating the database, but the instances seemed too random.

  • Ron KnightsRon Knights Posts: 1,805

    In 20+ years of using DAZ products, I haven't encountered any Duplicate Formulas. Am I lucky, or what?! It seems this problem has existed for decades. Any such problem should be easily fixed in a very short time. What's going on here.

  • Richard HaseltineRichard Haseltine Posts: 102,730
    edited May 2021

    Ron Knights said:

    In 20+ years of using DAZ products, I haven't encountered any Duplicate Formulas. Am I lucky, or what?! It seems this problem has existed for decades. Any such problem should be easily fixed in a very short time. What's going on here.

    Most of the time it's a clash between two properties with the same name - usually two characters from different stores. The issue doesn't show up unless both are installed. it doesn't affect older figures (if a similar issue happened in Victoria 4 the second morph injected would simply replace the first) and it obviously does require two characters with the same name - since Genesis 8 has already had a longer life than previous Genesis generations it's more likely to crop up - and it also depends on there being ERC links. Fixing the issue is a matter of renaming one or both of the character properties involved.

    Post edited by Richard Haseltine on
  • ArtiniArtini Posts: 9,722
    edited May 2021

    Ron Knights said:

    In 20+ years of using DAZ products, I haven't encountered any Duplicate Formulas. Am I lucky, or what?! It seems this problem has existed for decades. Any such problem should be easily fixed in a very short time. What's going on here.

    You have probably bought a very limited amount of Daz 3D products - that's why you have not encountered any such problems.

    I have already bought over 27 thousands of items only from Daz 3D store (and counting) so it is easier to get such errors.

     

    Post edited by Artini on
Sign In or Register to comment.