Loading...
> >

'Export User Data' from Content DB Maintenance: What is User Data ? Export to where?

3dcheapskate3dcheapskate Posts: 2,720
edited August 2019 in Daz Studio Discussion

 Edit: Some salient points that I've discovered while posting to this thread.

1) User data seems to include anything that has the <Userwords> tag in its metadata. (Content Library (context menu) > Content DB Editor > Assets > Userwords)

2) User data is exported in XML format to a file named 'UserData_1.dsx' in the /Runtime/Support folder of your first mapped DAZ content directory (Edit > Preferences >Content Library > Content Directory Manager >DAZ Studio Formats)

3) It's not just the end user who adds user data. A (very) small number of DAZ products include 'user' data in their metadata (the ones I found were  DAZ_3D_13176_Default_Lights_and_Shaders_for_DAZ_Studio.dsx and DAZ_3D_16281_Victoria_6.dsx - I think this is possibly a mistake that slipped past QA ? ).See this post.

 

 

(Original post starts here)

N.B. I'm using DS4.8

My understanding of metadata, the content database, and smart content, as they relate to products, is (in brief) as follows:

- The metadata for a product is contained in the /Runtime/Support/<productname>.dsx XML file. If the product doesn'thave this file then it has no metadata.
- If you install a DAZ product via DIM the metadata from this dsx  file is automatically added to the content database.
- If you use 'Manual Download', after copying the contents of the ZIP file to a content directory you need to manually add the /Runtime/Support/<productname>.dsx XML file to the metadata queue,and DAZ Studio will add it the content database.
- A product is 'smart content' if its metadata includes compatibility information.

So I'm guessing that 'User Data' is anything in the content database that did not originate from a /Runtime/Support/<productname>.dsx XML file.
Is that correct ?

I tried exporting the userdata to see what it contains (see screenshot) but I have no idea where it went - it didn't ask me for a path/fileto export to, and it didn't tell me where it was exported to.
So where did it go ?

 

Related threads: Any way to view the metadata queue before processing it ?

export.jpg
620 x 580 - 63K
Post edited by 3dcheapskate on
«1

Comments

  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019

    N.B. I'm using DS4.8

    I'm also guessing that if I select an item that has no metadata from the content library, right click and select Show In > Smart Content Files/Products, than that'll be 'User Data' ?

    When I added the example item above it immediately appeared in Smart Content > Files.
    However it vanished when I selected 'All' from the Smart Content >Files category list, and didn't appear even if I used the filter. It didn't appear in 'Unassigned' either.

    But if I use the Content DB Editor then it's there.
    So what's going on ?

    showin.jpg
    425 x 610 - 40K
    crap.jpg
    735 x 670 - 64K
    Post edited by 3dcheapskate on
  • MelanieLMelanieL Posts: 7,464

    Where does User Data go?  It lands in Runtime/Support/Userdata1.dsx (or some such) in probably the top content library as defined in DS content manager (as far as I remember). You can look in the log file and it will tell you exactly where it has put it.

  • MelanieLMelanieL Posts: 7,464

    And yes, I think anything you categorize yourself will be considered User Data  by DS - also any scenes, scene subsets, presets, etc that you have saved, so it is probably quite a rag-tag set of stuff.

    If you use Smart Content and select the "Product" called "Local User" then you should see everything there, I think.

  • Richard HaseltineRichard Haseltine Posts: 102,291

    User data is your own categorisation, that has not been made part of a product. It gets exported to a series of userdata*.dsx fiels in the /Runtime/Support/ folder in the first Daz Studio format content directory.

  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019

    Thanks MelanieL/Richard.

    I can confirm (using DS4.8) that it gets exported to '/Runtime/Support/UserData_1.dsx' in the first DS Content Manager listed content directory. Further exports simply overwrite that file.

     

    So what's in that user data file then?

    - It contains metadata for scenes that I saved ages ago, scenes that no longer exist.
    - It contains metadata for the product I'm currently finalizing. Which surprises me because I created the FBA/FBX support file pair, added that to the metadata queue, and let DS import it.
    - It also contains some DAZ product stuff for which I'm 99.9% certain I haven't added any user data (annotated screenshot collage below)


    However, it does not contain any reference to the four hands that I tried to manually add to Smart Content a couple of posts back.
    In fact those four hand items don't appear anywhere in Smart Content when I look from within DS.
    Is that perhaps because even though I'd used Show In > Smart Content, I hadn't given it a category, so there was no metadata at all ?
    (But in that case I'd expect it to appear in Smart Content under 'Unassigned' which it doesn't)

    usermyarse.jpg
    820 x 775 - 248K
    Post edited by 3dcheapskate on
  • 3dcheapskate3dcheapskate Posts: 2,720

    So how do I delete all this spurious 'User Data' from the content database ?

  • JonnyRayJonnyRay Posts: 1,744
    3dcheapskate said:

    So how do I delete all this spurious 'User Data' from the content database ?

    First question: Why do you care? I know as a long time database guy myself I don't like the idea of spurious data in my database, but do you think it's causing any problems that NEED to be remediated?

    Second question: Have you tried any of the options that are already available in the Content DB Maintenace menu? Descriptions added according to MY understanding of what the functions do, people with more knowledge can certainly correct me!

    • Remove Un-Mapped Base Paths and Orphans - Should remove base paths that you previously had defined, but deleted and weren't purged from the Content DB
    • Remove Orphaned File References - Should remove references to files which no longer exist
    • Consolidate File References - Should consolidate references where there are more than one reference to the same file
    • Condense Database - Typically purges unused space in a database, making access of the remaining data faster. Especially useful after large amounts of data have been removed.

    Other than being a "data purist", which as I mentioned I can understand, I'm not sure that having unused 'User Data' should cause any issues.

  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019
    JonnyRay said:
    3dcheapskate said:

    So how do I delete all this spurious 'User Data' from the content database ?

    First question: Why do you care? .... do you think it's causing any problems that NEED to be remediated?

    I think it might be the causes of some problems I'm having. Simply deleting the spurious user data would be the easiest way to confirm/refute that.
     

    JonnyRay said:
    Second question: Have you tried any of the options that are already available in the Content DB Maintenace menu? Descriptions added according to MY understanding of what the functions do, people with more knowledge can certainly correct me!

    Yes, the two 'Remove' options were the first things I tried (on the assumption that they'd remove references to stuff that no longer exists). But I still have user data for scenes that no longer exist, and those scenes still appear in my Smart Content. So 'Remove' didn't remove what I thought they'd remove.

     

    JonnyRay said:
    ...Other than being a "data purist", which as I mentioned I can understand, I'm not sure that having unused 'User Data' should cause any issues.

    I'd prefer to be certain. If I can remove the user data that I think might be causing the problem then I can confirm that it either was or wasn't the culprit.

    Post edited by 3dcheapskate on
  • JonnyRayJonnyRay Posts: 1,744
    edited August 2019

    So, here is what I would do to test this theory.

    1. Shut down both Studio and DIM (for belt an suspenders safety, make sure the Postgress SQL tasks are shutdown)
    2. Copy the content database folder somewhere as a backup
    3. Load Studio and use the "Reset Database" option from the content db maintenance - as I understand it, this should re-initialize the tables as if the database were brand new
    4. You may need to then follow that with "Re-Import Metadata", I'm not sure if it happens automatically with the reset, at least you should Process Metadata Queue
    5. Export the User Data and look to see if it looks clean now
    6. If it still looks wonky, then shutdown Studio and delete the content database folder, this will certainly force Studio to rebuild the database from scratch
    7. If at any time you feel like you're losing Smart Content functionality, I'd stop everything, restore your copied database folder, and look for a Plan C
    Post edited by JonnyRay on
  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019

    Thanks for the suggestion - I'm still trying to see if I can simply remove the spurious user data (and in that way actually understand how the user data, Categories, Products, Smart Content,and metadata are tied together). Resetting the database is still a last resort.

    First thing this morning I ran the other two basic tidy-up options from Content DB Maintenace, i.e. 'Consolidate File References' and 'Condense Database' - Products, Categories and Smart Content all vanished, and I had  to restart DS to see them again.
    I then exported User Data only to find that all the spurious user data was still there.

    I then somehow managed to remove all the spurious user data for scenes that I'd saved to the desktop and deleted ages ago - I spent a while trying to back track and work out what I'd done.
    It turned out to be quite straightforward - simply 'Remove Reference' from within the Categories section of the Content Library.

    So that resolves how to get rid of a particular subset of the user data, i.e.
    "- It contains metadata for scenes that I saved ages ago"

    But still leaves:
    "- It contains metadata for the product I'm currently finalizing..."   ...looking at the exported userdata again I think that this is actually the correct metadata,set up by DS running the FBA/FBX from the queue
    "- It also contains some DAZ product stuff for which I'm 99.9% certain I haven't added any user data"   ...I noticed that the Content DB Editor contains a 'UserWords' tab, so if I can find the items. Edit: that seemed logical, but... see next post.

     

    Edit: Reminder to self- don'tforgetabout the Poser hands andheads !

     

    remove.jpg
    984 x 1058 - 132K
    Post edited by 3dcheapskate on
  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019

    I also thought  I'd worked out why some DAZ products appear in the exported user data - they all have this <Userwords> section.

    I was wrong.

    I'd spotted a 'Userwords' tab in the Content DB Editor, so I made what appeared to be a logical deduction: these items appear in the exported user data because they have Userwords. Therefore, by deleting the Userwords they should no longer appear in the exported user data.

     

    But when I exported the user data after doing thisI got a big surprise. The four items still appear in the exporteduser data, along with the 17 other items from this product.

    And none of them contain any <Userwords>.

    And in addition, I can't find any support files for this product inany of my mapped <ContentDirectory>/Runtime/Support folders.
    (I was going to say that DS has deleted them, but I can't be certain that they were there before. But the metadata must have come from somewhere originally)

    logical.jpg
    1278 x 908 - 231K
    counterintuitiveorwhat.jpg
    1763 x 438 - 322K
    Post edited by 3dcheapskate on
  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019

    Try one of the other groups of stuff that appears in the exported metadata - the freebie product I'm trying to finalize the metadata for (which is what got me into this whole palaver in the first place ! ;o)

    First try simply deleting the metadata via the Content Library > Products > Delete Installed Metadata

    Restart DS4.8, delete the UserData_1.dsx file, and export user data.

    All the metadata for my 'product' is still there.

    crying

     

    deletemetadata.jpg
    376 x 335 - 26K
    Post edited by 3dcheapskate on
  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019

    Purge Unlisted doesn't help - everything's still here when  export user data...

     

    Neither does sync... - everything's still here when I export user data...

     

    How about reset ?

    (I didn't bother with JonnyRay's wisely recommended steps 1 and 2 because I was rather fed up by now and didn't really care... devil
    Also, if steps 1 and 2 were really necessary DAZ Studio would tell you, wouldn't it...)

    Well,at least that did something more or less predictable,and didn't take long - less than a minute I'd say.

    No more Smart Content !laugh

    And if I export user data it's very quick and it doesn't appear to create a <userdatawhatever>.dsx file. So maybe there's no longer any user data in the database ? Or it's exporting the data somewhere else - but I still have the same content directories listed in the same order in my preferences.

    purge.jpg
    1027 x 681 - 139K
    sync.jpg
    756 x 714 - 109K
    reset.jpg
    406 x 462 - 37K
    Post edited by 3dcheapskate on
  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019

    So after completely destroying resetting my Content Database it might be an idea to put some stuff back.

    But I'm going to wait a bit and have a think. There's a couple of references to userdata on that reimport dialog.

    reimport.jpg
    582 x 846 - 91K
    Post edited by 3dcheapskate on
  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019

    ...instead of that I simply copied the Support files (FBA/FBX/JPG) for my product to the C:\Users\UserName\AppData\Roaming\DAZ 3D\Studio4\RunOncefolder and restarted DS4.8.

    For the first time really I paid attention to the tickboxes regarding user data, and unticked them

    My product is now the only thing in Smart Content.

    And if I try to export user data it's very quick and it doesn't appear to create a <userdatawhatever>.dsx file.

    Maybe I need to intentionally create some user data and check that it gets exported ? Just to make sure that the userdata fileis going where I expect it to be. (Thinks: a popup saying"no user data found, no user data file written" would have been helpful)

    Started re-importing metadata for DAZ products, and the UserData_1.dsx file has reappeared in the expected place. 
    <Userwords>in the following two metadata files are theculprit:
    - DAZ_3D_13176_Default_Lights_and_Shaders_for_DAZ_Studio.dsx
    - DAZ_3D_16281_Victoria_6.dsx

     

    Edit: I mustn't forget that this proves the support files for my freebie actually work as I expected them to.

    reimportonstartup.jpg
    420 x 439 - 21K
    Post edited by 3dcheapskate on
  • JonnyRayJonnyRay Posts: 1,744

    Thank you for capturing all of this information. I always enjoy learning about what may (or may not) be working "under the covers" of the software tools I use.

    I was also looking at my Studio yesterday, I'm running 4.11. The options on the Content DB Maintenance are different. For instance the "remove orphans" functions aren't there at all.

    Out of curiousity, I did run the "Compress Database" tool and my content DB folders went from 2.47 GB down to 1.87 GB; so a pretty significant reduction in size.

    There's also a database log text file in there which Postgress seems to use for logging DB activity. Reading that highlighted why I see some metadata failing to process (duplicate product IDs).

  • 3dcheapskate3dcheapskate Posts: 2,720

    I've learnt one big lesson from this - Content Library context menu > Content DB Maintenance > Reset Database is my friend ! wink

  • JonnyRayJonnyRay Posts: 1,744
    Unless you have done a lot of custom categories. Then you're going to cry.
  • DannisWritesDannisWrites Posts: 51

    Thanks for this discussion! I suppose, does this mean I can get a handy list for a certain scene of every asset I used in that scene? If so, hot dog! I've been making little text files for each listing the assets so I can go find licensing in the readme's. This would simplify my licensing tracking immensely!!

  • 3dcheapskate3dcheapskate Posts: 2,720
    DanniStories said:

    Thanks for this discussion! I suppose, does this mean I can get a handy list for a certain scene of every asset I used in that scene? If so, hot dog! I've been making little text files for each listing the assets so I can go find licensing in the readme's. This would simplify my licensing tracking immensely!!

    Unfortunately, I don't think so - although I believe that there are other ways to create such a list of all assets used in a scene (mCasual may possibly have a script to do that, so it's worth checking/asking - https://sites.google.com/site/mcasualsdazscripts/)

    The exported user data referred to in this thread is that which exists in the DS Content Database - nothing to do with any individual scene.

  • 3dcheapskate3dcheapskate Posts: 2,720
    edited August 2019
    JonnyRay said:
    Unless you have done a lot of custom categories. Then you're going to cry.

    Indeed crying

    Unless...

    ...I wonder if custom categories are considered as user data ? If they are then you could export the user data first, check it (just tobe sure!), reset,and then reimport.

    P.S. I did say theat I found that resetwas my friend, not everybody's... wink

    Post edited by 3dcheapskate on
  • ChoholeChohole Posts: 33,604
    DanniStories said:

    Thanks for this discussion! I suppose, does this mean I can get a handy list for a certain scene of every asset I used in that scene? If so, hot dog! I've been making little text files for each listing the assets so I can go find licensing in the readme's. This would simplify my licensing tracking immensely!!

    http://docs.daz3d.com/doku.php/public/software/dazstudio/4/referenceguide/scripting/api_reference/samples/metadata/list_products_used/start

  • cm152335cm152335 Posts: 421
    Chohole said:
    DanniStories said:

    Thanks for this discussion! I suppose, does this mean I can get a handy list for a certain scene of every asset I used in that scene? If so, hot dog! I've been making little text files for each listing the assets so I can go find licensing in the readme's. This would simplify my licensing tracking immensely!!

    http://docs.daz3d.com/doku.php/public/software/dazstudio/4/referenceguide/scripting/api_reference/samples/metadata/list_products_used/start

    link error, 
    link have different target that it show,, 
     

  • ChoholeChohole Posts: 33,604

    Sorry about that     http://docs.daz3d.com/doku.php/public/software/dazstudio/4/referenceguide/scripting/api_reference/samples/metadata/list_products_used/start

  • JonnyRayJonnyRay Posts: 1,744
    3dcheapskate said:

    ...I wonder if custom categories are considered as user data ? If they are then you could export the user data first, check it (just tobe sure!), reset,and then reimport.

    Is there an "Import User Data"? I don't recall seeing one. Granted, it seems a bit silly that there wouldn't be, but I can't recall it on the database maintenance menu.

    3dcheapskate said:

    P.S. I did say theat I found that resetwas my friend, not everybody's... wink

    Okay, Brer Rabbit. cheeky

  • DannisWritesDannisWrites Posts: 51

    @Chohole, 

    Meant to answer you back in August [shows how far behind I am in EVERYTHING]. I'll have to look into that when I have a techy day [sigh]. I wish someone would write a script for listing all the assets in a scene. I may try it someday but my mind is mush today [and most days, unfortunately]. Thanks very much for the info! I'll save that link!!

  • DannisWritesDannisWrites Posts: 51

    @3DCheapskate,

    I did check over at mcasual, and he doesn't have a script for that, but I downloaded a whole bunch of his useful scripts...and months later, got them finally installed in my new content dir!! Thanks for suggesting his site. I'd been over there before many times, but had to figure out how to install the scripts...everything takes me forever, but I eventually get there :D

  • Richard HaseltineRichard Haseltine Posts: 102,291
    DanniStories said:

    @Chohole, 

    Meant to answer you back in August [shows how far behind I am in EVERYTHING]. I'll have to look into that when I have a techy day [sigh]. I wish someone would write a script for listing all the assets in a scene. I may try it someday but my mind is mush today [and most days, unfortunately]. Thanks very much for the info! I'll save that link!!

    http://docs.daz3d.com/doku.php/public/software/dazstudio/4/referenceguide/scripting/api_reference/samples/metadata/list_products_used/start

  • DannisWritesDannisWrites Posts: 51

    Thanks very much!!

  • mavantemavante Posts: 734

    Can anyone tell me why I have both a "UserData_1.dsx" and a "UserData_2.dsx" in my [Daz 3D Library]/Runtime/Support folder?

Sign In or Register to comment.