3Delight Laboratory Thread: tips, questions, experiments

13132343637100

Comments

  • mjc1016mjc1016 Posts: 15,001

    But the interesting thing...the CPU percentage drop seems to match the time increase wowie had...using a 1/3 less CPU results in a 1/3 longer render?

  • mjc1016mjc1016 Posts: 15,001
    edited November 2015

    Well...double post by not doing anything...

    Post edited by mjc1016 on
  • wowiewowie Posts: 2,029
    edited November 2015
    mjc1016 said:

    Hey this may explain the slower renders...

    http://www.daz3d.com/forums/discussion/comment/937792/#Comment_937792

    I suppose that there are ways to include a throttle.

    Nope. That's not it. All my CPU cores are utilized properly - no throttling. I re-render with the task manager in view. Anyway if it did throttle, it would be doing it with just AO too.

    Post edited by wowie on
  • So is it different in the standalone or with another GI shader?

  • ZarconDeeGrissomZarconDeeGrissom Posts: 5,412
    edited November 2015
    wowie said:
    mjc1016 said:

    Hey this may explain the slower renders...

    http://www.daz3d.com/forums/discussion/comment/937792/#Comment_937792

    I suppose that there are ways to include a throttle.

    Nope. That's not it. All my CPU cores are utilized properly - no throttling. I re-render with the task manager in view. Anyway if it did throttle, it would be doing it with just AO too.

    I had experienced almost the exact same CPU usage once a very long time ago, lol. Stupid simple reason as well.

    I had the "Staph of render hell" in a scene, doing a regular render (not progressive mode). The instant the render worked it's way down to where the staph was, the reflection (or specular) caused the 'Buckets' to go on hold till the reflected image was calculated. Because 3DL was told to only work on X many buckets, and they were all waiting on the reflection to be calculated, Everything came to a grinding crawl, lol. Easiest solution, bump up the number of "Buckets", and the CPU kept busy, lol.

    http://www.daz3d.com/forums/discussion/comment/914462/#Comment_914462

    Also, it may be some debugging code in the beta between studio and 3DL, tho that is mostly a guess at this point. You say ALL cores are in use, so there doing something for something somewhere (even if it is just an endless loop of add 1 to itself till it gets to 4,294,967,295. lol).

    Post edited by ZarconDeeGrissom on
  • wowiewowie Posts: 2,029
    edited November 2015

    So is it different in the standalone or with another GI shader?

    Hmm,

    Have you checked if the output RIB with DS4.9 beta rendered correctly? I tried rendering two RIB files, one with your script and one with the non-scripted renderer. I keep getting 'Fatal Errors' with the RIB from your script. The RIB from the non scripted renderer works fine, except for the usual different interface version warnings. I'm still using the old script though.

    Got around to testing your DelightGIHDRI. Same scene, just substituted UE2 with it. Render time with 4.7 is 24 minutes 13.62 seconds (pretty much the default settings, only color, intensity and max trace distance was changed). Here's the render time with 4.9 beta - 25 minutes 5.49 seconds. Slower, but not as much as with UE2. I'm using 64 samples with your DelightGIHDRI, and 128 samples with UE2. So the results are not directly comparable to each other.

    While the shader used for GI might have something to do with it, it's still slower than what's expected (which is faster than previous 3delight builds).

     

    Post edited by wowie on
  • wowiewowie Posts: 2,029

    Trying out Crytek's version of the famous Sponza set. Still very rough, only spent about 1 hour setting up (importing the OBJ, loading textures, applying shaders and positioning/setting lights). I've turned off all the extra bits (banner, ornaments etc) to get a closer look to the original model. Still very rough. Just AO and linear point lights, so no indirect lighting and bounced rays.

    Sponza.jpg
    1894 x 1065 - 1M
    Sponza2.jpg
    1894 x 1065 - 1M
  • Mustakettu85Mustakettu85 Posts: 2,933
    edited November 2015
    wowie said:

    Have you checked if the output RIB with DS4.9 beta rendered correctly? I tried rendering two RIB files, one with your script and one with the non-scripted renderer. I keep getting 'Fatal Errors' with the RIB from your script. The RIB from the non scripted renderer works fine, except for the usual different interface version warnings. I'm still using the old script though.

    Yes, everything renders fine. In my case, it's the same 34 seconds in the standalone as it was in 4.8. (there is a single figure in the scene, but it has GI and glossy reflections)

    The "fatal error" with RIBs is due to the Standard Example not doing certain things. I have fixed it since. You should make sure you install the newest alpha scripts into the DAZStudio4 Public Build folder in your ProgramFiles and use the new scripts to generate the RIBs, then everything will be OK.

    Here is a Windows shortcut for running ribdepends to do "collect and localise" on the RIB (replace the paths with yours):

    "C:\Program Files\3Delight\bin\ribdepends.exe" -package "C:\Users\Forever\Documents\!_3D models\RIB\AnnC" "C:\Users\Forever\Documents\!_3D models\RIB\Anna_C.rib"

    Don't close DS before you run this; the folder into which the files are copied _must_ exist.

    wowie said:

    Got around to testing your DelightGIHDRI. Same scene, just substituted UE2 with it. Render time with 4.7 is 24 minutes 13.62 seconds (pretty much the default settings, only color, intensity and max trace distance was changed). Here's the render time with 4.9 beta - 25 minutes 5.49 seconds. Slower, but not as much as with UE2. I'm using 64 samples with your DelightGIHDRI, and 128 samples with UE2. So the results are not directly comparable to each other.

    While the shader used for GI might have something to do with it, it's still slower than what's expected (which is faster than previous 3delight builds).

    Thank you for reporting these results. Now we need to find out the rendertime with my shader in the standalone.

     

    PS When you render a RIB exported with the newest alpha script, it should create an HTML file with stats in the same folder. This will give you precise render time; look for a line like this, the "real" is what we need, but it's in seconds:

    Total process times (user/sys/real) : 160.66 / 6.06 / 34.29

    Post edited by Mustakettu85 on
  • I had experienced almost the exact same CPU usage once a very long time ago, lol. Stupid simple reason as well.

    I had the "Staph of render hell" in a scene, doing a regular render (not progressive mode). The instant the render worked it's way down to where the staph was, the reflection (or specular) caused the 'Buckets' to go on hold till the reflected image was calculated. Because 3DL was told to only work on X many buckets, and they were all waiting on the reflection to be calculated, Everything came to a grinding crawl, lol. Easiest solution, bump up the number of "Buckets", and the CPU kept busy, lol.

    Zarcon, thank you for posting this. It confirms my suspicion. I was doing some stuff in 4.8 today and noticed the CPU load kept dropping down even then. I guess I should dial down the default bucket size in my scripts to avoid that.

    As for "debugging code", my first thought was that it shouldn't really be the case... but then, I thought about how IPR is implemented - the "world" is regenerated on the fly, correct? So yeah, there may be something happening in the API between the renderer and DS even when there is no IPR.

  • wowiewowie Posts: 2,029
    edited November 2015

    Zarcon, thank you for posting this. It confirms my suspicion. I was doing some stuff in 4.8 today and noticed the CPU load kept dropping down even then. I guess I should dial down the default bucket size in my scripts to avoid that.

    16 is a good number. I ran a comparison of several bucket sizes a long time ago (8,16, up to 128) and found 16 to be the most robust value. Of course, I don't generally use shaders other than US2 (which has an ray trace depth override for reflection). If you're using DAZ Studio default shaders, calculating reflections and interreflections can cause huge slowdowns.

    Post edited by wowie on
  • Wowie, I just edited the previous post to show where to get rendertime when rendering RIBs coming out of my scripts =)

  • wowiewowie Posts: 2,029

    Wowie, I just edited the previous post to show where to get rendertime when rendering RIBs coming out of my scripts =)

    Ah thanks. I usually just call up the Info on i-display.

  • Thank you Kettu and wowie, I'm glad I was able to add two 'farthings' (something smaller then a penny) of experience to your incredible wealth of knowledge. (Yes that's a complement)

    I found a good note to keep in mind, that I over looked yesterday day. Studio 4.9 Beta is only 3DL 12.0.27, the one just before that reflection optimization thing. While the nuts and bolts is still way over my head, I can understand how an optimization can potential change the performance of things, and I was curious to test that regarding reflections vs number of buckets.

    Your 'Stand alone' may be a newer version of 3DL then what is in the Studio beta. I don't know if a license allows you to get the latest updates or not, I haven't dug that far into 3DL stand alone, yet.

    http://3delight.atlassian.net/wiki/display/3DSP/Changelog

  • Zarcon, that optimisation is for a specific shading model that pre-built DS shaders don't currently use... only certain custom ones =) The free standalone currently is a lower build than that (12.0.19). Ah, I can only wish I were a real CG artist who could justify getting the premium service...
  • wowiewowie Posts: 2,029
    edited November 2015

    Thank you Kettu and wowie, I'm glad I was able to add two 'farthings' (something smaller then a penny) of experience to your incredible wealth of knowledge. (Yes that's a complement)

    Moi? Hey, I'm just a sucker with a lump in my throat. :D

    More seriously though, most of the improvements in 3delight are generally unused by existing 3delight shaders for DAZ Studio. They don't use the more advanced and more physically plausible specular (GGX/GTR) or raytraced subsurface scattering. Only Kettu's shader implements those. Don't know about mjc1016's though.

    Post edited by wowie on
  • mjc1016mjc1016 Posts: 15,001
    wowie said:

    Thank you Kettu and wowie, I'm glad I was able to add two 'farthings' (something smaller then a penny) of experience to your incredible wealth of knowledge. (Yes that's a complement)

    Moi? Hey, I'm just a sucker with a lump in my throat. :D

    More seriously though, most of the improvements in 3delight are generally unused by existing 3delight shaders for DAZ Studio. They don't use the more advanced and more physically plausible specular (GGX/GTR) or raytraced subsurface scattering. Only Kettu's shader implements those. Don't know about mjc1016's though.

    Yes, my SSS shader uses raytracing, a couple others I'm playing with use the new features. 

  • wowiewowie Posts: 2,029
    edited November 2015

    Re-rendering the scene with the standalone. Results with just UE2 AO is 15 min 24 seconds. Results with UE2 IDL and your latest script is 50 min 52 seconds. Weird.

    BTW Kettu, do you know how to copy a color from a channel (diffuse color) to another (translucency or 2nd diffuse)? Without overwriting the already present texture maps.

    Post edited by wowie on
  • Weird indeed. And what would the time be for the standalone and my shader? And also for the 'bounce light' UE2 mode?

    I'm not on the computer right now, otherwise I would've given you the exact code right away. Basically if you look at my copy diffuse map script, it's done the same way but you are not using the map-handling methods but the colour-handling ones (don't remember the exact syntax by heart). If you're in a hurry, you can look the methods for those classes up in the zipped DS3 doc file here : docs.daz3d.com/doku.php/public/software/dazstudio/3/start . Or I'll get back to this tomorrow.

  • mjc1016mjc1016 Posts: 15,001

    Just the color?

    I don't know of an automated way of doing it...but I click on the color bar to bring up the color picker and slide the color into one of the 'user' slots and then manually change it in the others.

    As to the times...the only other thing I can think of is that the UE2 is using custom code that isn't taking advantage of or not really compatible with the newer features of 3DL.

  • wowiewowie Posts: 2,029
    edited November 2015

    Weird indeed. And what would the time be for the standalone and my shader? And also for the 'bounce light' UE2 mode?

    I'm not on the computer right now, otherwise I would've given you the exact code right away. Basically if you look at my copy diffuse map script, it's done the same way but you are not using the map-handling methods but the colour-handling ones (don't remember the exact syntax by heart). If you're in a hurry, you can look the methods for those classes up in the zipped DS3 doc file here : docs.daz3d.com/doku.php/public/software/dazstudio/3/start . Or I'll get back to this tomorrow.

    Well, I've just finished rendering a .RIB exported from 4.7 to the standalone. Render times is 49 minutes and 57 seconds. I would have to say, it's tied to the renderer version. I imagine rendering with your DelightGIHDRI would produce roughly the same times as before. By the look of things, I think I'm going to stick to 4.7 for now.

    Already reported the problem to the help desk. Maybe they can channel the report back to 3delight devs.

    mjc1016 said:

    Just the color?

    I don't know of an automated way of doing it...but I click on the color bar to bring up the color picker and slide the color into one of the 'user' slots and then manually change it in the others.

    Already have a script to copy textures from casual. I've modified it now so it will copy diffuse textures from the 1st channel to the translucence color and 2nd diffuse channel, plus opacity maps to the 2nd diffuse strength slot, and of course bump/displacement from the 1st to the 2nd channels.

    Problem is that some props/materials uses colors and not textures. So when I apply my preset (which doesn't override diffuse colors), you still have to manually enter the color values for the 2nd diffuse (and translucene color). The most ideal would be a script that copies the color values but clamps them to a min value of 32 and a max value of 160, but that requires some proper normalization math.

    mjc1016 said:

    As to the times...the only other thing I can think of is that the UE2 is using custom code that isn't taking advantage of or not really compatible with the newer features of 3DL.

    Well, there's two things there. Either someone change 3delight or UE2. Both really out of our hands.

    Post edited by wowie on
  • And do IDL and bounce modes of UE2 give the same result? 

  • wowiewowie Posts: 2,029
    edited November 2015

    And do IDL and bounce modes of UE2 give the same result? 

    Didn't test bounce GI mode. Generally bounce GI is going to take longer, so I don't really want to test it, even for the sake of completion. 30 minutes on my machine means about 1 and half hour on Zarcon's spec. 50 minutes is about 2 and half hours.

    On a side note, your DelightGIHDRI light already performs better than UE2. :D Your shaders and lights are definitely the one to use in the future.

    Thanks for the link to the DAZ Script SDK. Figured it out now. Well, except for the clamp min/max and normalization bit. :) But hey, let the user fiddle with that.

    Post edited by wowie on
  • Thank you =) I guess there are two things behind performance: a) using contemporary  shadeops; b) each new install of my shaders requires compiling anew because they come as shader builder files. So I thought - as a last resort, even though version 12 does not throw a warning about shaders compiled in 11, maybe UE2 could perform better if recompiled? Takeo's script could be pointed at the beta folder and run: www.daz3d.com/forums/discussion/23867/free-script-to-recompile-daz-shaders/p1

    You're welcome =) Clamping is doable with the max and min methods of the Math object, I think (with several if-then conditions): docs.daz3d.com/doku.php/public/software/dazstudio/4/referenceguide/scripting/api_reference/object_index/math

  • mjc1016mjc1016 Posts: 15,001

     So I thought - as a last resort, even though version 12 does not throw a warning about shaders compiled in 11, maybe UE2 could perform better if recompiled? Takeo's script could be pointed at the beta folder and run: www.daz3d.com/forums/discussion/23867/free-script-to-recompile-daz-shaders/p1

    That might be something to check...but what if it's a custom loop/depricated code? 

    (I'm getting the feeling it might be a custom loop...isn't/wasn't OF more of a straight Renderman user?)

     

  • wowiewowie Posts: 2,029
    edited November 2015

    You're welcome =) Clamping is doable with the max and min methods of the Math object, I think (with several if-then conditions): docs.daz3d.com/doku.php/public/software/dazstudio/4/referenceguide/scripting/api_reference/object_index/math

    Yeah. I've been thinking about that. I'd probably go with clamping to a min value but normalizing the existing color so the highest color value is 160. That's what I generally do anyway via the color picker, though at this moment it's done manually. Maybe with a HSV scheme, by making sure value should be no more than 64.

    I'll try recompiling the shaders.

    Edit: No. Doesn't change anything.

    Post edited by wowie on
  • ZarconDeeGrissomZarconDeeGrissom Posts: 5,412
    edited November 2015

    I don't think UE2 is, well, a high priority item to keep optimizing for newer CPUs and stuff. Isn't that a Daz thing now, like Hexagon? Then again, it may be the only reason I've noticed things render faster with newer Studio versions (4.5 threw 4.8, haven't tried 4.9 yet).

    Going back to Studio 4.7 days (it may have been 4.6), when I tried a few deferent modes for mimicking ambient light. UE2 appeared to be best on the Occlusion modes for giving me something for test renders (setting up scenes) without making me wait days in some cases (on the old 4core CPU). The newer 8core CPU is better (even tho it wiped out my BBC documentary DRM validations, grrrr), The newer 3DL in the newer Studio versions is better, and I haven't felt like torchering my self that much with testing it again, lol. The attached settings (surfer guy tutorial map used) appear to be the best for setting up scenes, anything more is just diminishing returns vs the excruciating wait setting up scenes (I do some times drop the shadow bias, sometimes).

    I spent the past day, looking at the SORH settings. It is almost ready, I just need to put some text on some renders, and hunt down the stuff to put into a zip. This thing indeed brings 3DL to it's knees (I haven't figured out Iray settings fot it yet). One of these renders took an hour, and when it got to 98% done, my CPU still dropped to 20% load, lol. To much cowbell this time, lol. Oh, and this cool one, I decided to not include the lights in the staff, as that was just for fun, and 4.8 uses deferent lights then was in former Studio versions.

    I know I missed comments somewhere, it's BC, Before Coffee.

    P.S. I'm just the new kid around, and half the stuff y'all do (shader mixer, making new surface/light shaders, etc), is still way beyond my skills. I'm still working on AoA/Omni surface setting basics, lol.

    UE2Settings_NoteToSelf_001.png
    535 x 695 - 29K
    Types_Clasic_r.png
    400 x 400 - 303K
    Types_Revised_notGoodr.png
    400 x 400 - 316K
    Types_Revised_r.png
    400 x 400 - 306K
    SORH_Promo_009_r.png
    800 x 800 - 825K
    Post edited by ZarconDeeGrissom on
  • wowiewowie Posts: 2,029
    edited November 2015

    I don't think UE2 is, well, a high priority item to keep optimizing for newer CPUs and stuff. Isn't that a Daz thing now, like Hexagon? Then again, it may be the only reason I've noticed things render faster with newer Studio versions (4.5 threw 4.8, haven't tried 4.9 yet).

    In perspective, outside of Kettu's lights and shaders, no one have come close to what omnifreaker's stuff have achieved. I firmly believe that all these years, what's keeping DAZ users tapping the full potential of 3delight is the interface we have to it. Layman's terms - it's the shaders, lights, unexposed parameters and lack of proper documentation.

    As I said before - bringing a knife to a gun fight. You can still win of course, but it takes special effort to do so. smiley

    Post edited by wowie on
  • wowie said:

    I don't think UE2 is, well, a high priority item to keep optimizing for newer CPUs and stuff. Isn't that a Daz thing now, like Hexagon? Then again, it may be the only reason I've noticed things render faster with newer Studio versions (4.5 threw 4.8, haven't tried 4.9 yet).

    In perspective, outside of Kettu's lights and shaders, no one have come close to what omnifreaker's stuff have achieved. I firmly believe that all these years, what's keeping DAZ users tapping the full potential of 3delight is the interface we have to it. Layman's terms - it's the shaders, lights, unexposed parameters and lack of proper documentation.

    As I said before - bringing a knife to a gun fight. You can still win of course, but it takes special effort to do so. smiley

    Understandable. In a way, I kind of with omnifreaker was still around improving things (optimizing, etc). I get the impression, that like so many other things, nothing has been done to some things in a very long time. And from the sounds of it, 'compiling' a shader is in a league beyond the ability of some, reducing there access to highly optimized routines to the far less efficient 'scripting' of sorts.

    Don't get me wrong, I know some specialized surface shaders can be made in Shader mixer that run circles around the more complicated compiled shaders. I't's just that some things (like rippling water) are so incredibly difficult to get it looking good without it taking days to render, lol. case and point, the spot render I'm waiting on now.

    SORH_DryrunTest_001.png
    1819 x 1010 - 1M
  • mjc1016mjc1016 Posts: 15,001
     

    Don't get me wrong, I know some specialized surface shaders can be made in Shader mixer that run circles around the more complicated compiled shaders.

    No...never going to happen.  While it may be true for the compiled shaders that are included with Studio, it isn't true in general.  A compiled shader will always out perform a Shader Mixer network.  And the more complicated the network the worse it is.  You would probably see a doubling or more in speed if the AoA shaders were compiled shaders as opposed to SM networks.

    And as to the complexity...again, it's back to the interface and how difficult it is to get the more complex shader code to be recognised and compiled by Shader Builder.  Without the support scripts, it is not worth bringing them in...and some of the more complex ones, while they will compile in Studio (there is another way to compile them, than SB) would need to have the support scripts manually created to make them useful for anything than a one shot, hard coded deal.  And the manual creation of those support scripts relies entirely on complete documentation...

    With what you are doing, things should be taking no more than an hour or two...even at fairly large image sizes.  Especially with an 8 core processor.  If that was the kind of performance to expect from 3DL, in general, it would have ceased to be one of the goto renderers for Hollywood, a long time ago, instead of being continually used and desired. 

  • wowiewowie Posts: 2,029
    mjc1016 said:

    No...never going to happen.  While it may be true for the compiled shaders that are included with Studio, it isn't true in general.  A compiled shader will always out perform a Shader Mixer network. 

    Well, I wouldn't say that exactly. If a Shader Mixer network is not doing complicated, complex stuff as a compiled shader, it might just render faster. But of course, it will have very limited use.

    I see Shader Mixer is a kind of rapid prototyping tool. Add a little bit of this and that, set rules how they interact with each other, the light or surface properties etc. Once you've got it all mapped out, take it to your TD and get him/her/them to make a comparable compiled shader.

    I do wish there was some nice way to get procedurals into a compiled shader, short of baking them. But i can live with that.

Sign In or Register to comment.