Library validation tool
Pickle Renderer
Posts: 236
Hi,
Is there a tool that will trawl through your entire content library looking for things that are "broken"? You know, checking references of other items exist, etc. I don't mind if it runs all night.
Daz 3D is part of 
Connect
DAZ Productions, Inc.
7533 S Center View Ct #4664
West Jordan, UT 84084
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
Not that I am aware of.. What do you mean by "broken"?? If something isn't working correctly, put in a support ticket. If you use DIM it shows you updates to products when they come out
I take it you are looking for a dependency checker? Are the textures installed for a scene or a character, do the morphs exist, that kind of thing?
If so, sadly not. Any tool that could do this would have to trawl through every single line of every single file in your library and then perform an existence check of any dependencies it found. Those dependencies might have their own dependencies and so on. Recursive purgatory. It might be possible to do but I think you would have to be prepared to wait all week for something like that to finish if you have even a moderately sized library.
If something doesn't work, is missing textures or morphs, or is in some other way utterly borked check the log. Fix it, reinstall it or report it. It would be much, much quicker than trying to pre-scan for errors!
Yea, dependency checker. That's what I'm looking for. I actually think it's quite an easy problem to solve with a few passes. Can also be made reasonably efficient.
It's an easy programming problem, sure. I'm a professional coder in C++ and C# (Unreal and Unity respectively) and I admire your optimism at the execution time of a script through even a moderately sized library on a magnetic disk though!
You could write something like this as a standalone tool with a compiled (fast) language because it's essentially a text scanner and a file checker. You could store checked and confirmed references internally so if they pop up again in another text file they could be marked as already confirmed but, quite honestly, I don't see the point of doing this.
1. Load something into Studio. Does it work? Yes? Awesome.
2. No? Argh! Check the log for errors. Reinstall as required. Fix obvious errors in PA supplied files. Is it fixed? Yes? Awesome.
3. Still not fixed? Report error to Daz or vendor.
You'll still have to do the above if your dependency checker flags errors so why go through the hassle. Just fix it as it pops up.
(Edited to add): You'd also need to call an external compression tool to decompress those files which have been saved with compression on and then delete those temporary files after they've been scanned and the references stored/checked. Still think this would be quick?!
Apologies if I have missed something that would make such a tool inherently useful - it's been a long day and I'm a little tired so I am genuinely open to an explanation as to why this would be a good addition to the arsenal. Sometimes you miss the obvious thing staring you in the face!
I'm also a software dev (C++, C#, Unreal, Industrial). I don't mind the runtime. I don't think you need an external decompression tool. It's just .zip format isn't it (in the libraries already). Will decompress in memory very quickly. In the past I've written a tool that does something similar across libraries of electronic component designs. The caching phase (basically generating a bunch of big hashtables) took a while. After that it was really fast.
When I look at all the threads (probably dozens of them if not hundreds) about character loading performance, often caused by library errors if not simply the size of the library, I think a validate feature in Daz itself or some other tool to do it would be very useful.
I actually know very little Daz Script because the documentation is woefully out of date and hopelessly incomplete. It's essentially Javascript with a lot of Daz functionality and it's the latter that it is missing from the (hah!) documentation. I do suspect you could do it in Studio's native script but I don't know enough about it to make any useful contribution. Basically, what I know of it I've learned from reading unencrypted files from other coders. I also know, from trial and error, that it's not that quick and compiled at runtime. You'd be better off treating this as a standalone application as I think we both know.
I see what you're driving at with the hash tables but, even armed with that database, what are you going to do about speeding up character loading times? Duplicate formulas won't even be on the radar of this tool and these are the major cause of loading time increases. Validated morphs won't flag as errors and if you have thousands of these perfectly viable files they load synchronously. It's all single-threaded.
When duplicates occur they are flagged for correction in real-time. Why scan in advance? I still don't see a reason for this kind of tool and I have no motivation to implement it but you could. You could submit it as a product for the store if you think there's a market for it. Perhaps I'm wrong and there's a real demand for this kind of thing but I think Studio already flags issues effectively - the only issue it doesn't flag is 'your morph libraries are getting too big'. It *is* a problem but it's not something an external tool can fix. Daz need to sort this problem themselves.
I thought you were looking for products where you had inadvertently deleted some dependencies. Or they weren't properly installed in the first place. If you're looking to speed up character loading times I think you're on to a hiding to nowhere but I look forward to you proving me wrong because, if you do, I will be first in line to buy your product!
There is an, admittedly, older tool from Dimension3D called File Reference Editor. Still works, but only one file at a time, and I don't see any options for batch operations, or to redirect the output.
I have something for checking single products, but it is not not memory efficient enough (or fast enough) to check most content libraries.
It may serve as a starting point for creating what you want, however it is written in Python and you would likely need to reimplement or make major changes to be able to do an entire library without running out of memory.
https://pypi.org/project/valdazpack/
I forgot about Content Checkup by @Totte (Code 66). It may be what you are looking for.
Yes, you can validate a library, but not your "main one", too many files, but a subset of it., or a product.