After 20+ years...

1235

Comments

  • almahiedraalmahiedra Posts: 1,353

     

    Chatjd said:

    This is not the Daz "supported" way to do it, I'm sure, but I manually edit the actual support dsx metadata file that contains the smart content information.  With a text editor (I use Notepad++), it is fairly quick to do search and replace changes depending on how many items there are in a product. 

    With some lines of code in any software for scripting this task can be automatized. I am using Rstudio and writing even a general custom category before installing. Unfortunately I can't read the content type info in each asset to do more detailed categorization, but when I can purchase Wizard content this problem will be solving.

     

     

  • Ron KnightsRon Knights Posts: 1,805

    I've encountered one huge problem with the search function in DIM. You can't search for all Genesis products. It will only find items with the word "Genesis" in the name. I've worked around it by searching for "5, 6, 7, 8," This will catch Victoria 5, Michael 6, etc. Unfortunately you also get anything with that number in the name.

  • Sensual ArtSensual Art Posts: 645
    GiGi_7 said:

     

    Chatjd said:

    This is not the Daz "supported" way to do it, I'm sure, but I manually edit the actual support dsx metadata file that contains the smart content information.  With a text editor (I use Notepad++), it is fairly quick to do search and replace changes depending on how many items there are in a product. 

    With some lines of code in any software for scripting this task can be automatized. I am using Rstudio and writing even a general custom category before installing. Unfortunately I can't read the content type info in each asset to do more detailed categorization, but when I can purchase Wizard content this problem will be solving.

    I picked up the wizard for nearly 85% off last month. Its ok for sensibly allocating default categories for 3rd party content (when they are missing), otherwise DS's built-in Content DB editor alone can do almost all the 'magic' stuff. Everytime I open the Content DB editor on 3rd party content it stares at me with an empty face mocking the failure of smart content.

    Wizard.jpg
    577 x 162 - 27K
  • brvsnbrvsn Posts: 213

    I've encountered one huge problem with the search function in DIM. You can't search for all Genesis products. It will only find items with the word "Genesis" in the name. I've worked around it by searching for "5, 6, 7, 8," This will catch Victoria 5, Michael 6, etc. Unfortunately you also get anything with that number in the name.

    there's a default filter however.. can't say if it works for you.. It's "Filter by Name ---> Daz 3D Figures" - all main generations are there..

  • Sensual ArtSensual Art Posts: 645
    Chatjd said:
    With ~20k DIM installed products (which probably equal out to 500k-100 million items), just scrolling through categories can slow down significantly because of the sheer amount.  Anything to put less burden on the database helps. 

    The database is a ticking time bomb. I have more that 1.8 million content entries for just 8k Connect installed products. That amounts to roughly 3.3 GB of metadata without any optimizations other than a few indexes. Whats more concerning is that only roughly above 25% of that is for user facing entries. The rest is just configuration overhead for support assets that could slow down categories or DB based content search.

    dzcms.png
    1161 x 256 - 38K
  • Ron KnightsRon Knights Posts: 1,805

    brvsn, wow! I never knew about this before! Thank you so much!

  • Richard HaseltineRichard Haseltine Posts: 102,907
    mrinal said:
    Chatjd said:
    With ~20k DIM installed products (which probably equal out to 500k-100 million items), just scrolling through categories can slow down significantly because of the sheer amount.  Anything to put less burden on the database helps. 

    The database is a ticking time bomb. I have more that 1.8 million content entries for just 8k Connect installed products. That amounts to roughly 3.3 GB of metadata without any optimizations other than a few indexes. Whats more concerning is that only roughly above 25% of that is for user facing entries. The rest is just configuration overhead for support assets that could slow down categories or DB based content search.

    Export your user data (using Content DB Maintenance, which can be launched from the Content Library option menu - the lined button in the top corner or right-click the tab) and you will be able to rebuild the database in case of corruption.

  • fixmypcmikefixmypcmike Posts: 19,613
    mrinal said:
    Chatjd said:
    With ~20k DIM installed products (which probably equal out to 500k-100 million items), just scrolling through categories can slow down significantly because of the sheer amount.  Anything to put less burden on the database helps. 

    The database is a ticking time bomb. I have more that 1.8 million content entries for just 8k Connect installed products. That amounts to roughly 3.3 GB of metadata without any optimizations other than a few indexes. Whats more concerning is that only roughly above 25% of that is for user facing entries. The rest is just configuration overhead for support assets that could slow down categories or DB based content search.

    The CMS database is nowhere near the limits of what PostgreSQL can handle:

    Limit   ValueMaximum Database Size       UnlimitedMaximum Table Size          32 TBMaximum Row Size            1.6 TBMaximum Field Size          1 GBMaximum Rows per Table      UnlimitedMaximum Columns per Table   250 - 1600 depending on column typesMaximum Indexes per Table   Unlimited

    I don't know for certain, but I would guess that the bottleneck in scrolling through categories is in displaying the graphics, not in querying the database.

  • Sensual ArtSensual Art Posts: 645
    mrinal said:
    Chatjd said:
    With ~20k DIM installed products (which probably equal out to 500k-100 million items), just scrolling through categories can slow down significantly because of the sheer amount.  Anything to put less burden on the database helps. 

    The database is a ticking time bomb. I have more that 1.8 million content entries for just 8k Connect installed products. That amounts to roughly 3.3 GB of metadata without any optimizations other than a few indexes. Whats more concerning is that only roughly above 25% of that is for user facing entries. The rest is just configuration overhead for support assets that could slow down categories or DB based content search.

    Export your user data (using Content DB Maintenance, which can be launched from the Content Library option menu - the lined button in the top corner or right-click the tab) and you will be able to rebuild the database in case of corruption.

    I have not created a single custom category so far, as I do not find the categorization approach to be scalable given the sheer amount of content and the risks and overhead involved in managing them. I rely solely on item search (DB based) to lookup content I need, other than characters, of course. Not that I find the Daz Studio's DB based search option very appealing given its lack of sorting and filtering(outside scenes/scripts) capabilities but I find it lesser of the two evils compared to the unproductive overhead of manually maintaining categories. Take for example Amazon, or basically any other e-commerce site which has catalog of millions of items. They do offer elaborate categorization but how many of us really navigate through those categories to lookup the items we need? Please enlighten me how I can efficiently wade through the 100s or even 1000s of search results in Daz Studio involving poses, materials, shaders etc when all I am looking for is just a prop (say a candle)?

    I am not worried about losing any custom category information in case of database malfunction/corruption but I am more worried about item level search getting impacted due to the bloat of non-userfacing content residing in the same tables/tablespace. I haven't tried with user tags as most of the time the default tags have proven sufficient. But if I ever did, that might be worth backing up.

  • HavosHavos Posts: 5,404

    I am always somewhat bemused at the efforts people go to organise their content. I simply install everything with DIM, and never move anything, nor bother with categories. 90% of the time I can find what I want using Smart Content, and the remaining 10% I just use the content library, and since I never moved anything then DIM tells me exactly where every file was placed. Even for non DAZ content, if I don't know where something was placed I just open the zip and can see exactly where I can find it, where as if I had moved it around potentially years before I would probably struggle to find it.

    I understand there is no perfect content management system, and everyone has to do what is best for them. However just because some people find Smart Content "useless", doesn't mean we all do. I personally find it very useful, and a massive time saver.

  • Sensual ArtSensual Art Posts: 645

    I prefer the Smart Content search more than the Content Library search since it is somewhat faceted. Very useful for searching accessories and props that are spread across multiple content packages under various categories. But again the limitation there is the 'facets' or categories in Smart Content do not display the number of results or provide any visual indication if they(or any of their children) contain the searched item. So one still has to rely on their instincts/luck to find an item under a category or risk navigating through the entire search results.

  • kyoto kidkyoto kid Posts: 41,260
    edited July 2020

    KyotoKid, I'm still trying to see when or how I can set up a surgery date. And I'm also trying to deal with a sleep doctor to get a new BiPap machine, etc. Our apartment building will replace its one elevator in July. I hope to avoid the discomfort that might cause me. Maybe I'll get the surgery after the new elevator is installed.

    I'm still working on the manual install of my DAZ Content. It feels so nice to know exactly where to find everything.

    ....makes good sense to wait until the lift is repaired as in the weeks immediately afterwards dealing with stairs is impossible (that friend of mine had his bed moved down and slept in the living room so everything would be on one level).

    Yeah, I'm fortunate that building I'm in has three lifts (one just a few steps from my unit's door) as I am on the top (5th) floor. 

    Whenever there is a fire alarm, which is usually set off by something very minor like someone burning toast or microwave popcorn (the individual alarms in each unit are hardwired into the entire building's alarm system so everyone has to evacuate), it locks off the lifts.  There have been many times with my stiff arthritic knees and hips that I will barely get out the the ground floor door to the pavement and the fire department has already has arrived and dealt with the situation (there is a station only a few blocks away).

    Yeah it's a lot of work initially doing a manual setup and creating up tour own library/runtime structure, but well worth it in the end.  Definitely make a back up of that drive when you are done. as it's no fun having to rebuild it all over again.  I nearly gave up on continuing with this after the last drive crash.

    Excuse any typos., had a serious crash of Chrome & Google last night that completely locked my system and wouldn't resolve even after the first restart (took nearly 90 min to deal with).  Consequently several extensions (like the browser spell check) were apparently corrupted and refuse to work anymore, so I have to copy/paste to Word to check for errors in anything I write including emails, forum, and on social media posts (the forum spell check here is useless as it makes really stupid suggestions that make autocorrect on my phone look "genius").

    [edited for typos]

    Post edited by kyoto kid on
  • Roman_K2Roman_K2 Posts: 1,253

    lol, you PAs are some of the most disorganized people I've ever come across.

    <Blink> ROTFL.!!! </Blink>

    My favorite (not!) is when they jump the queue by making up a new category -- like maybe putting an "s" on the end of a word -- or worse, they've set it up so that their John Henry is a category in and of itself. angry

  • Roman_K2Roman_K2 Posts: 1,253
    marble said:

    I've never used Smart Content so I've removed that tab from my workspace.

    Roger that!

  • Ron KnightsRon Knights Posts: 1,805

    kyoto kid, the way things are going, I'd be surprised if I could get the knee replacement surgery this month. I think our building management somehow got a few waivers on things like how many elevators they had. Oh, and they've kept changing the date of when the elevator might be installed. From my experience, things will always go wrong. I need to get out soon and do some major grocery shopping. I got food stamps from the county government, and can't let them go to waste.

  • SnowSultanSnowSultan Posts: 3,648
    edited July 2020
    Roman_K2 said:

    lol, you PAs are some of the most disorganized people I've ever come across.

    <Blink> ROTFL.!!! </Blink>

    My favorite (not!) is when they jump the queue by making up a new category -- like maybe putting an "s" on the end of a word -- or worse, they've set it up so that their John Henry is a category in and of itself. angry

    Oh it used to be worse, a long time ago vendors would put exclamation marks and spaces in the product names to bump them to the top of listings, not to mention burying them in subfolders a mile deep. I remember having to go in and fix directories with names like "! ! ! 3Ddude/Realistic/Figures/People/Michael 4/Clothing/Sci_Fi/SpaceySuit"   indecision

    Things have definitely improved - but I'm still sorting by hand.

    Post edited by SnowSultan on
  • 3Diva3Diva Posts: 11,749
    edited July 2020
    brvsn said:

    I've encountered one huge problem with the search function in DIM. You can't search for all Genesis products. It will only find items with the word "Genesis" in the name. I've worked around it by searching for "5, 6, 7, 8," This will catch Victoria 5, Michael 6, etc. Unfortunately you also get anything with that number in the name.

    there's a default filter however.. can't say if it works for you.. It's "Filter by Name ---> Daz 3D Figures" - all main generations are there..

    Oh my goodness, I didn't know of this function! Clicking the little spyglass icon opens up a ton of search parameters! Very cool - thank you!

    Post edited by 3Diva on
  • Ron KnightsRon Knights Posts: 1,805

    I remember many years ago when Poser couldn't handle all our content. Some people used utilities to change the extension of the character files so they'd go into a different Poser library. And then Hogwarden created a Utility that allowed us to switch between different banks of content. That led to a huge problem where Poser locked up. That was because many references were wrong. So we got Correct Reference Pro... All that happened around 1999.

  • PerttiAPerttiA Posts: 10,024

    I remember many years ago when Poser couldn't handle all our content. Some people used utilities to change the extension of the character files so they'd go into a different Poser library. And then Hogwarden created a Utility that allowed us to switch between different banks of content. That led to a huge problem where Poser locked up. That was because many references were wrong. So we got Correct Reference Pro... All that happened around 1999.

    Don't get me started on Poser and it's content management - DS is from another (way better) planet, even with the PA:s that cannot see the "bigger picture".

  • 3Diva3Diva Posts: 11,749
    edited July 2020
    PerttiA said:

    I remember many years ago when Poser couldn't handle all our content. Some people used utilities to change the extension of the character files so they'd go into a different Poser library. And then Hogwarden created a Utility that allowed us to switch between different banks of content. That led to a huge problem where Poser locked up. That was because many references were wrong. So we got Correct Reference Pro... All that happened around 1999.

    Don't get me started on Poser and it's content management

    LOL Yeah... As much as people like to complain about the folder/categorization system in Daz Studio, it is WAY better and MUCH easier to find stuff than the nightmare that is the poser "categorization" system.

    Post edited by 3Diva on
  • almahiedraalmahiedra Posts: 1,353
    edited July 2020
    mrinal said:
    GiGi_7 said:

     

    Chatjd said:

     

    I picked up the wizard for nearly 85% off last month. Its ok for sensibly allocating default categories for 3rd party content (when they are missing), otherwise DS's built-in Content DB editor alone can do almost all the 'magic' stuff. Everytime I open the Content DB editor on 3rd party content it stares at me with an empty face mocking the failure of smart content.

    A very good price! 

    I live in Venezuela, then move money to foreign is almost impossible without a bank account in other country, and people with accounts charge to much for a simple transaction (even friends), in my first attempt to buy WContent I had to give up. Probably with the new accounts in Panamá for venezuelans I can move some dollars for daz store and wait for discounts.

    Post edited by almahiedra on
  • Richard HaseltineRichard Haseltine Posts: 102,907

    I remember many years ago when Poser couldn't handle all our content. Some people used utilities to change the extension of the character files so they'd go into a different Poser library. And then Hogwarden created a Utility that allowed us to switch between different banks of content. That led to a huge problem where Poser locked up. That was because many references were wrong. So we got Correct Reference Pro... All that happened around 1999.

    Daz has never, at least in theory, allowed that for folder names. For file names, yes, so that the most important settings are first, but not for folder names.

  • almahiedraalmahiedra Posts: 1,353
    edited July 2020

    The 2 paradigms for folders in DS. I want a mix of the pros, why we can't get something like separated folders for products and content for other stores allowed?

     

    DAZ FS.png
    1591 x 691 - 108K
    Post edited by almahiedra on
  • Richard HaseltineRichard Haseltine Posts: 102,907
    mrinal said:

    I prefer the Smart Content search more than the Content Library search since it is somewhat faceted. Very useful for searching accessories and props that are spread across multiple content packages under various categories. But again the limitation there is the 'facets' or categories in Smart Content do not display the number of results or provide any visual indication if they(or any of their children) contain the searched item. So one still has to rely on their instincts/luck to find an item under a category or risk navigating through the entire search results.

    The number of items found is listed just to the right of the filter box.

  • Ron KnightsRon Knights Posts: 1,805
    edited July 2020

    Richard, I am not sure what you mean. I'm talking about Poser 4 content, back around 1999 or 2000. I was a beta tester for PBooost and Correct Referance Pro

    Post edited by Ron Knights on
  • gitika1gitika1 Posts: 948

     

    1. Have both DS and DIM open.

    2. Stop the CMS in DS.

    3., Install some items with DIM (I find 6 at a time is a good manageable number for me). 

    4. Start the CMS in DS.

    5. "Scan Known Directories for Content"

    6. Sort the results into your desired categories

    7. Re-import metadata (if you have DIM sorting installed items by most recent install date, it's easy to just select the most recent items).

    8. Repeat as needed.

    (I should note that I'm really anal-retentive, so I look at the folder structure in the zip before installing, and submit a bug report if it's incorrect or if anything is missing after DIM installs it.  I also look at all the newly-installed items after re-importing metadata and make sure they all have the correct tag at the top right for what type of item them are.  It's not necessary to do all that, I'm just OCD about it.)

    So, you are finding the folder for the product under Daz Studio formats and categorizing from there?  When I first starting categorizing, I would find the newly acquired product in the Products tab, select an asset and then choose Show Asset in mapped folder.  Lately, this process has failed because some products don't show in the Products Tab or Smart Content tab.  (an example: Wild Poses for Felidae Lion)  They can be found via the folders under Daz Studio formats, but then DS labels the product as Local User.

    Skipping the product list would probably save time, as it sometimes is extremely slow loading the products for a letter.

    Are your user file back-ups multiple and large?  I find that mine are.

    If the correct tabs are missing, are you editing the dsx?  I'm going to need to learn how to do that.

  • Sensual ArtSensual Art Posts: 645
    mrinal said:

    I prefer the Smart Content search more than the Content Library search since it is somewhat faceted. Very useful for searching accessories and props that are spread across multiple content packages under various categories. But again the limitation there is the 'facets' or categories in Smart Content do not display the number of results or provide any visual indication if they(or any of their children) contain the searched item. So one still has to rely on their instincts/luck to find an item under a category or risk navigating through the entire search results.

    The number of items found is listed just to the right of the filter box.

    I am not referring to the total number of items in search results. I am referring to the breakdown and display of that number besides individual categories if they contain the search results. For example, if I search for a "hat" in Smart Content the total results may be 100s or even 1000s since it includes all user facing assets (with metadata) including poses, scripts and materials. But the category list on the left does not display which categories contain hats from the search results or how many hat entries I can expect from the category before I navigate to it. So if I am not sure if that hat I am looking for is a figure, prop, accessory or wardrobe then I would have to traverse through each of these categories in Smart Content to find the right hat.

    Is asking for a count to be displayed against each category for search results in Smart Content too much to expect?

  • Sensual ArtSensual Art Posts: 645
    mrinal said:
    Chatjd said:
    With ~20k DIM installed products (which probably equal out to 500k-100 million items), just scrolling through categories can slow down significantly because of the sheer amount.  Anything to put less burden on the database helps. 

    The database is a ticking time bomb. I have more that 1.8 million content entries for just 8k Connect installed products. That amounts to roughly 3.3 GB of metadata without any optimizations other than a few indexes. Whats more concerning is that only roughly above 25% of that is for user facing entries. The rest is just configuration overhead for support assets that could slow down categories or DB based content search.

    The CMS database is nowhere near the limits of what PostgreSQL can handle:

    Limit   ValueMaximum Database Size       UnlimitedMaximum Table Size          32 TBMaximum Row Size            1.6 TBMaximum Field Size          1 GBMaximum Rows per Table      UnlimitedMaximum Columns per Table   250 - 1600 depending on column typesMaximum Indexes per Table   Unlimited

    I don't know for certain, but I would guess that the bottleneck in scrolling through categories is in displaying the graphics, not in querying the database.

    A few missing pieces of information from the limits that you have specified:

    • The above limits do not specify if the tests were run on commodity harware such as desktops and laptops or on specially provisioned servers and equipments which are designed specifically for benchmarking database systems.
    • The above limits do not specify if any advanced database tuning methodolgies (such as sharding/partitioning) were used to support such large datasets.
    • The above limits do not specify the complexity of the sampled datasets i.e. whether those rows constituted real-world data or some programmatically generated data.
    • The above limits do not provide any sort of benchmarks with regards to performance (such as query execution time) on commodity hardware - desktops and laptop, when dealing with large real-world datasets (millions of rows) of actual application data.

    Due to the absence of the above mentioned information it can be logically concluded that those supposed limits may be contextual to certain environments and should not be interpreted as an assurance or guarantee for universal usage. So it would be inadequate to justify the scalabilty of the Daz CMS[*] just on the basis of the specified limits of PostgreSQL.

    [*] Daz CMS refers to the copy of PostgreSQL (including its data and configuration) that is used by Daz software suite on users desktop.

  • fixmypcmikefixmypcmike Posts: 19,613
    gitika1 said:

     

    1. Have both DS and DIM open.

    2. Stop the CMS in DS.

    3., Install some items with DIM (I find 6 at a time is a good manageable number for me). 

    4. Start the CMS in DS.

    5. "Scan Known Directories for Content"

    6. Sort the results into your desired categories

    7. Re-import metadata (if you have DIM sorting installed items by most recent install date, it's easy to just select the most recent items).

    8. Repeat as needed.

    (I should note that I'm really anal-retentive, so I look at the folder structure in the zip before installing, and submit a bug report if it's incorrect or if anything is missing after DIM installs it.  I also look at all the newly-installed items after re-importing metadata and make sure they all have the correct tag at the top right for what type of item them are.  It's not necessary to do all that, I'm just OCD about it.)

    So, you are finding the folder for the product under Daz Studio formats and categorizing from there?  When I first starting categorizing, I would find the newly acquired product in the Products tab, select an asset and then choose Show Asset in mapped folder.  Lately, this process has failed because some products don't show in the Products Tab or Smart Content tab.  (an example: Wild Poses for Felidae Lion)  They can be found via the folders under Daz Studio formats, but then DS labels the product as Local User.

    Skipping the product list would probably save time, as it sometimes is extremely slow loading the products for a letter.

    Are your user file back-ups multiple and large?  I find that mine are.

    If the correct tabs are missing, are you editing the dsx?  I'm going to need to learn how to do that.

    since I stop the CMS before installing, "scan Known directories" puts all the new stuff in Default > Scan Results, and I sort it from there.

    My userdata backups don't seem too large -- currently there are 3 files totaling about 100MB.  They're ascii files, so they compress by 98% for storage.  Every so often I back up the actual ContentCluster (with DIM and DS closed and no postgres processes runnng) -- that's about 2GB and compresses about 85%.

    If I find metadata or other errors I usually just submit a bug report.

  • nicsttnicstt Posts: 11,715
    Roman_K2 said:
    marble said:

    I've never used Smart Content so I've removed that tab from my workspace.

    Roger that!

    +1

Sign In or Register to comment.