Any way to reduce size of overall Daz library without just uninstalling content?

brettnuckles

So I keep my Daz library on an external HDD because I don't have the space on my computer's internal hard drive. However I've been feeling frustrated lately with how insanely slowly stuff loads into DAZ -- it takes a WHILE to load a figure, guessing like 2-8 minutes on the crazy end.

My solution was to grab an external SSD to put my library on, hoping I can load content much faster from that, but unfortunately my library is just slightly too large to fit on the drive I bought. Is there any way to reduce the overall library size a bit without just starting to uninstall a bunch of content? I checked my Daz library stats in WinDirStat and noticed like 2/3 or more of my library is just textures. Maybe there's a way to reduce those texture sizes so they'll still work but at a slightly lower quality (I seriously wouldn't mind, I mostly use Daz as reference for 2D illustrations)? Sorry, this isn't my area of technical expertise at all.

Penname

I deleted a lot of 3Delight textures, and don't load them for new purchases, since I don't use them.  That helped some.  The newer products are massive space hogs for sure.

brettnuckles

Penname said:

I deleted a lot of 3Delight textures, and don't load them for new purchases, since I don't use them.  That helped some.  The newer products are massive space hogs for sure.

Interesting!  I'm not honestly sure how that would work. Would I have to go through my whole library manually and delete 3Delight textures one by one, making sure that each product has working iRay textures before doing so? If so I'm not sure that's feasible, my library is pretty massive. I assume doing a system-wide search for "3Delight" and deleteing everything I find would be a pretty bad idea? Sorry for the dumb questions, the way Daz files work has always seemed like a real convoluted mess to me so I've never really dug into understanding them.

NorthOf45

The biggest source of delay for loading figures is all the morphs that need to be initialized that really slows things down, if you have a lot of them.

The bitmaps can be used for any render engine, they are not specific to 3DL or Iray, and they are usually the same set of files (don't know about Filament or PBR). Maybe you deleted the presets, but that is miniscule compared to the size of the bitmaps. Large bitmaps will not slow down the process that much, and they have to be loaded anyway, so unless you want to resample them, there's not much you can do about it.

(There are utilities for Studio that will optimize and re-sample, but won't delete the orignals, so it ends up taking more space. They were intended to make the scene fit into the VRAM for rendering, not for loading speed. Easier to use some external graphic utility like IrfanView to re-sample then delete the originals. You can always back them up or just retrieve a new copy if you want them again.)

Also, you can have more than one library, and more than one disc, for all your content. That can be configured in Studio with Content Directory Manager and DIM (if you use that to install).

brettnuckles

NorthOf45 said:

The biggest source of delay for loading figures is all the morphs that need to be initialized that really slows things down, if you have a lot of them.

 

Thanks for the input! I do have a pretty crazy amount of morph packs, maybe I will uninstall some because I have a bunch I don't use often. Did not know that was a primary source of loading delay so this is very useful! Makes perfect sense. No wonder my figures started to load slower and slower over time -- I was accumulating more morph packs!

I did see someone in another older thread suggest using the "compress contents to save space" option in the Windows folder properties, which they said would make my library take up less space without messing with loading times (much?). Anyone know anything about whether that's accurate?

NorthOf45

brettnuckles said:

NorthOf45 said:

The biggest source of delay for loading figures is all the morphs that need to be initialized that really slows things down, if you have a lot of them.

 

Thanks for the input! I do have a pretty crazy amount of morph packs, maybe I will uninstall some because I have a bunch I don't use often. Did not know that was a primary source of loading delay so this is very useful! Makes perfect sense. No wonder my figures started to load slower and slower over time -- I was accumulating more morph packs!

I did see someone in another older thread suggest using the "compress contents to save space" option in the Windows folder properties, which they said would make my library take up less space without messing with loading times (much?). Anyone know anything about whether that's accurate?

NTFS compression will benefit the text files and optimize the cluster usage, but will have little or no effect on the bitmaps, as they are already compressed about as much as they can be. Access time won't be much affected, and you might get 10-15% more disc space, but it's not like you will double your capacity. Depends on how much you expect to add to your library. Keep it in perspective. If you've spent a car's worth on content, don't skimp on the tires.

PerttiA

You will not gain much by moving your content to SSD, maybe leave your old HDD to store the textures but put your geometry (characters\morphs) on the SSD

I have my single content folder spread out to four or five drives, data files on SSD and textures on USB HD, DS still sees just one (standard) content folder

PerttiA

NorthOf45 said:

brettnuckles said:

NorthOf45 said:

The biggest source of delay for loading figures is all the morphs that need to be initialized that really slows things down, if you have a lot of them.

 

Thanks for the input! I do have a pretty crazy amount of morph packs, maybe I will uninstall some because I have a bunch I don't use often. Did not know that was a primary source of loading delay so this is very useful! Makes perfect sense. No wonder my figures started to load slower and slower over time -- I was accumulating more morph packs!

I did see someone in another older thread suggest using the "compress contents to save space" option in the Windows folder properties, which they said would make my library take up less space without messing with loading times (much?). Anyone know anything about whether that's accurate?

NTFS compression will benefit the text files and optimize the cluster usage, but will have little or no effect on the bitmaps, as they are already compressed about as much as they can be. Access time won't be much affected, and you might get 10-15% more disc space, but it's not like you will double your capacity. Depends on how much you expect to add to your library. Keep it in perspective. If you've spent a car's worth on content, don't skimp on the tires.

I wouldn't start using compression for complete disks... When the smelly stuff hits the fan, you are way deep in it...

nicstt

PerttiA said:

You will not gain much by moving your content to SSD, maybe leave your old HDD to store the textures but put your geometry (characters\morphs) on the SSD

I have my single content folder spread out to four or five drives, data files on SSD and textures on USB HD, DS still sees just one (standard) content folder

There will be some improvement, how much and what one considers 'much' is open to interpretation. I've been using SSDs for everything but backups for years now, and I certainly saw an improvement - although that was prior to G8.

brettnuckles

PerttiA said:

You will not gain much by moving your content to SSD, maybe leave your old HDD to store the textures but put your geometry (characters\morphs) on the SSD

I have my single content folder spread out to four or five drives, data files on SSD and textures on USB HD, DS still sees just one (standard) content folder

So is it possible that splitting my content this way would allow the figures themselves to load a bit faster? If so, could someone point me to some resources that would help me understand how to split up different parts of my library across different drives?

After reading through some other threads, I think I honestly will need to go through and uninstall a bunch  of my characters and morphs -- I have accumulated a totally unnecessary amount of content, and much of it is rarely used... I wish there was another solution, as I'm a bit of a digital hoarder and I like to have infinite possibilities at my fingertips, haha.

PerttiA

brettnuckles said:

PerttiA said:

You will not gain much by moving your content to SSD, maybe leave your old HDD to store the textures but put your geometry (characters\morphs) on the SSD

I have my single content folder spread out to four or five drives, data files on SSD and textures on USB HD, DS still sees just one (standard) content folder

So is it possible that splitting my content this way would allow the figures themselves to load a bit faster? If so, could someone point me to some resources that would help me understand how to split up different parts of my library across different drives?

After reading through some other threads, I think I honestly will need to go through and uninstall a bunch  of my characters and morphs -- I have accumulated a totally unnecessary amount of content, and much of it is rarely used... I wish there was another solution!

 The ...\Data\ folder in your content library is the important one, it holds the geometry for all the models and morphs. Textures are stored in ...\Runtime\Textures\ and they need a lot more space but are only read when actually needed.

You can redirect the contents of the Data folder to another drive by;

1.) Mounting another drive to that folder with standard Windows tools (Computer Management\Disk Management), when the drive becomes essentially your Data folder, preserving the original folder structure..

2.) Using SysInternals Junction, which can be downloaded from MS https://docs.microsoft.com/en-us/sysinternals/downloads/junction you can redirect any folder to any other drive or folder without breaking the original folder structure.

I have been using both methods for at least 3 years without having any problems with them and the best part in both is that the drives are still standard and the files can also be accessed through their assigned drive letters.

brettnuckles

PerttiA said:

brettnuckles said:

PerttiA said:

You will not gain much by moving your content to SSD, maybe leave your old HDD to store the textures but put your geometry (characters\morphs) on the SSD

I have my single content folder spread out to four or five drives, data files on SSD and textures on USB HD, DS still sees just one (standard) content folder

So is it possible that splitting my content this way would allow the figures themselves to load a bit faster? If so, could someone point me to some resources that would help me understand how to split up different parts of my library across different drives?

After reading through some other threads, I think I honestly will need to go through and uninstall a bunch  of my characters and morphs -- I have accumulated a totally unnecessary amount of content, and much of it is rarely used... I wish there was another solution!

 The ...\Data\ folder in your content library is the important one, it holds the geometry for all the models and morphs. Textures are stored in ...\Runtime\Textures\ and they need a lot more space but are only read when actually needed.

You can redirect the contents of the Data folder to another drive by;

1.) Mounting another drive to that folder with standard Windows tools (Computer Management\Disk Management), when the drive becomes essentially your Data folder, preserving the original folder structure..

2.) Using SysInternals Junction, which can be downloaded from MS https://docs.microsoft.com/en-us/sysinternals/downloads/junction you can redirect any folder to any other drive or folder without breaking the original folder structure.

I have been using both methods for at least 3 years without having any problems with them and the best part in both is that the drives are still standard and the files can also be accessed through their assigned drive letters.

Ooh, I was expecting this to be more of a pain in the neck than that! I did not realize you could mount a drive to function as a folder on a different drive! Neat! I will give this a shot, thanks!

Actually, would you recommend one of these methods over the other?

PerttiA

brettnuckles said:

PerttiA said:

brettnuckles said:

PerttiA said:

You will not gain much by moving your content to SSD, maybe leave your old HDD to store the textures but put your geometry (characters\morphs) on the SSD

I have my single content folder spread out to four or five drives, data files on SSD and textures on USB HD, DS still sees just one (standard) content folder

So is it possible that splitting my content this way would allow the figures themselves to load a bit faster? If so, could someone point me to some resources that would help me understand how to split up different parts of my library across different drives?

After reading through some other threads, I think I honestly will need to go through and uninstall a bunch  of my characters and morphs -- I have accumulated a totally unnecessary amount of content, and much of it is rarely used... I wish there was another solution!

 The ...\Data\ folder in your content library is the important one, it holds the geometry for all the models and morphs. Textures are stored in ...\Runtime\Textures\ and they need a lot more space but are only read when actually needed.

You can redirect the contents of the Data folder to another drive by;

1.) Mounting another drive to that folder with standard Windows tools (Computer Management\Disk Management), when the drive becomes essentially your Data folder, preserving the original folder structure..

2.) Using SysInternals Junction, which can be downloaded from MS https://docs.microsoft.com/en-us/sysinternals/downloads/junction you can redirect any folder to any other drive or folder without breaking the original folder structure.

I have been using both methods for at least 3 years without having any problems with them and the best part in both is that the drives are still standard and the files can also be accessed through their assigned drive letters.

Ooh, I was expecting this to be more of a pain in the neck than that! I did not realize you could mount a drive to function as a folder on a different drive! Neat! I will give this a shot, thanks!

Actually, would you recommend one of these methods over the other?

The first one is fine in your case.

I started using the second one when I needed to move individual overgrown texture folders to other drives that still had space on them, this can't be done with the first method.

PS. Move the contents of Data folder to the root folder of your new drive before mounting it to the original Data folder.

thenoobducky

Seperate your contents into seperate folder, keep the commonly used on fast drive and rarely used on slow drive. Character, morphs, expression, poses, and clothing with morphs have impact on loading speed but it is more CPU bound than HDD bound. Material/props/other figures that is not genesis dont have much impact and thus can be put on slower file.

Drive speed dont have much impact on load time. large amount of the loading time is spend on creating erc formula from morph files. Daz cache them by default on your C drive which hopefully will be an ssd. SSD will make daz more responsive when loading things like icon images and geometrys, but unless your external drive is busy doing other stuff it wont be much slower.

The real solution to slow loading time is to make daz selectively adding stuff as base directory. You can either manually seperate contents or make a script/program to automatically do that which is what I am working on right now.

 

BlueSirius

There is a script around that will go through a scene and grab everything it uses - I sometimes take a finished scene to my 2nd machine to render each frame of the timeline. This script grabs the textures, morphs, props everything to a smaller subset content library on my portable drive.

Look for Daz3D Scene exporter (Standalone) - V2 update by MCphylyss on ShareCG.

But this can also be used to make a slim content library version. .Suppose you extracted for example all the scene you are working on then pile these into a new content library you can then selectively stop looking at the main content library. When you want a real speed boost you have a content library only with assets used in your main scenes. When your shopping for assets to add to a scene have a beta version of Daz mapped to the bulk library and run as a 2nd session. If your working in Beta turn this around and have a release version that can see all the librarys but keep the beta blinkered to assets that have been used.

brettnuckles

BlueSirius said:

There is a script around that will go through a scene and grab everything it uses - I sometimes take a finished scene to my 2nd machine to render each frame of the timeline. This script grabs the textures, morphs, props everything to a smaller subset content library on my portable drive.

Look for Daz3D Scene exporter (Standalone) - V2 update by MCphylyss on ShareCG.

But this can also be used to make a slim content library version. .Suppose you extracted for example all the scene you are working on then pile these into a new content library you can then selectively stop looking at the main content library. When you want a real speed boost you have a content library only with assets used in your main scenes. When your shopping for assets to add to a scene have a beta version of Daz mapped to the bulk library and run as a 2nd session. If your working in Beta turn this around and have a release version that can see all the librarys but keep the beta blinkered to assets that have been used.

Oh, cool! I was wondering if there was any easy way to separate out specific products into a smaller library, and this should do. I use Daz as reference for 2D art primarily, so sometimes I just want a figure -- ANY figure! -- to throw into a pose real quick and it annoys me to no end having to wait 3 mintues for something to load! 

So I gave this scene exporter a shot, but I'm not fully sure how it works. I loaded in some characters and hit export to another folder, and it did indeed seem to build the new library of content just from that scene. Then I go into my Daz content directory manager and unhook the old content folder, and hook the new tiny one. However, now I'm left with a library full of broken links -- the whole database of my content shows up in Smart Content, but they're all broken links of course. And what's more, even if I search for the specific models that I exported into the new tiny library, they still say the content is missing and can't load them.

Interestingly, If I try to open the .Duf file I exported from, it DOES load the figures, but with no textures whatsoever...

Is there any documentation or could someone help me out here? Thank you!!

cm152335

here my tips:

i have severals libraries (internal or external)
since the first year i use a "separate" library strategy 
the "creation" part and the "assembly" part
-
as example:
I have all characters in a unique library named  CH-LIBRARY (no clothes or props just characters and their materials)
in another libray i have only HAIRS
for enviroments (houses, rooms and places) another libray named ROOMS 
depend if i create or assembly i change, add or remove path in "Content Directory manager"
when i want to create a new character only the CH-Library is in list of Content Directory
once character is created (and saved) i added next needed library (hair, clothes, poses and so)
in this way the application run faster, smother due to using "less memory" and less "hard disk access R/W"
-
the only pain in this way, i didn't use DM for install but i unzip manualy content in their related libraries
and the "Smart Content" is not always complete!
i have more tnan 12 separates libraries but only three at once are listed in Content directory manager,
t takes less than 2 minutes to set them and make you happy for hours 

BlueSirius

Brettnuckles - in so far as documentation all I know of with this script is what the author put on Sharecg. I take scenes to this other computer and they load up fine using the duf file it makes. Normally I use the exe version.

As the scenes have the same data structure as my content library I sometimes copy them over my content library on the second machine such that products I have not installed there get the bits and pieces used in any scenes I take over to that machine.

Another thing I do to avoid being stifled by slow loads is Multitask. I have Beta Daz and Release Daz running normally as well as Gimp and Hexagon. My content manager on one version of Daz3D is very blinkered so it loads fast. While I am rendering or doing anything with a wait I tab to the other Daz to look for assets I want or make a morph I will need etc. Basically anytime I need to wait I just flip to the other and do something.

 

CM152335 - as per what you say yes I pretty much abandon Smart Content and just use Content Library as I am also hiding content when not used by my present phase of work.