[Name change] Issues with extremely long Iray render times rendering on the GPU

kyoto kidkyoto kid Posts: 41,197
edited August 2021 in Daz Studio Discussion

[Changed the name to reflect the  issue at hand]

...I added a couple IES lights from Marshian's Horror And Action Lights For Figures And Vehicles to a scene I have been working on and noticed that while they made for a much better lighting solution, Unfortunately the render time shot up significantly than compared the same scene with just a normal Photometric point and spot light .  The scene is an interior setting though I removed most of what I could that wasn't seen in the camera frame   There are 9 mesh lights affecting scene as well, 4 of which are very low intensity  

The original version took about 45 min to render and was GPU based (older Maxwell Titan-X with the latest driver). I monitored performance using the MSI Afterburner utility to make sure VRAM and RAM usage limits were not exceeded during the process  VRAM use was about 4.3 GB (out of 12 GB) and RAM usage was just over 12 GB (out of 24)  Task Manager indicated the Daz programme was using about 9.6 GB. 

The version of Daz I am working in is the 4.12.0.47 Beta which has been extremely stable and reliable.

When I replaced the normal Photometric spot and point lights with the lES versions, at 87% convergence and nearly 20,000 iterations there was still some noise.  I cancelled the operation, but saved what had rendered up to that point.

This makes me wonder if IES profiles are more demanding on resources than normal emissive and Photometric lights.

The finished .,jpgs are attached below, the first one without the IES lights and the second, is with them.

 

Leela Groznek MultiKey jacket.jpg
700 x 900 - 333K
Leela multikey IES test.jpg
933 x 1200 - 608K
Post edited by kyoto kid on

Comments

  • FishtalesFishtales Posts: 6,161

    The IES Profile only tells the light how it should shine, if that makes sense, but shouldn't change the intensity unless it is built into the profile too. The second image looks as if something else has changed. There is a stronger light hitting the front of the girl and the light shining on the control panel has changed colour from white to purple. I would also lower the camera until the two bright spots are out of shot as I have found that when they are in shot it lengthens the render times.

  • charlescharles Posts: 849
    kyoto kid said:

    ...I added a couple IES lights from Marshian's Horror And Action Lights For Figures And Vehicles to a scene I have been working on and noticed that while they made for a much better lighting solution, Unfortunately the render time shot up significantly than compared the same scene with just a normal Photometric point and spot light .  The scene is an interior setting though I removed most of what I could that wasn't seen in the camera frame   There are 9 mesh lights affecting scene as well, 4 of which are very low intensity  

    The original version took about 45 min to render and was GPU based (older Maxwell Titan-X with the latest driver). I monitored performance using the MSI Afterburner utility to make sure VRAM and RAM usage limits were not exceeded during the process  VRAM use was about 4.3 GB (out of 12 GB) and RAM usage was just over 12 GB (out of 24)  Task Manager indicated the Daz programme was using about 9.6 GB. 

    The version of Daz I am working in is the 4.12.0.47 Beta which has been extremely stable and reliable.

    When I replaced the normal Photometric spot and point lights with the lES versions, at 87% convergence and nearly 20,000 iterations there was still some noise.  I cancelled the operation, but saved what had rendered up to that point.

    This makes me wonder if IES profiles are more demanding on resources than normal emissive and Photometric lights.

    The finished .,jpgs are attached below, the first one without the IES lights and the second, is with them.

     

    Not from my experience with IES. I almost always use IES over point anymore.
  • kyoto kidkyoto kid Posts: 41,197
    edited July 2021

    ....yeah I toned down the surrounding lights as the character seemed to be a bit lost in the background 

    The one on the character is behind and above the camera already while the one for the drone is in front of the drone  Both use point lights "out of the box", not spotlights.  The intensity of the one on the character. originally was very high (200K lumens for a headlamp) and I dialled it back quite a bit (100 lumens).  As to the light on  the panel (5 lumens) I wanted it to look more like a UV scan than a normal electric torch.

    The nice feature of these lights is they are easier to aim (hence The lines from the light to the target in the OpenGL screenshot below).

    The camera headlamp is turned on just for the purpose of producing screenshot, otherwise it would be black.

     

     

    multi key ss.jpg
    1452 x 996 - 822K
    Post edited by kyoto kid on
  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ...update: 

    I thought I found the culprit, I had the Post Denoiser enabled and set to start at around 2500 iterations. I used that setting for an earlier render tests of this same scene before changing the lighting as finished renders still had a lot of noise (on average the tests ran for a total of about 2,700 iterations before completing)  So that explains the super long render time as for most of the process it was applying the filter.  

    Again, at least that's what I thought.  

    I turned it off and started rendering again (leaving the iteration maximum where it was set) and the processes seemed to go a lot smoother so I turned in expecting to see it finished. Well woke up in the middle of the night, decided to check it and it was at 80% with at just under 18,000 iterations and taking about 5 minutes per 1% of convergence  There was also an odd statement about the albedo of one of the shaders where it apparently has stalled for some time (I changed shaders on the door and door frame to more of a matte quality metal rather than the glossy one I had used which did cut down on the fireflies seen in the previous attempt).  I let it go for a while longer then cancelled it at 85% after 2 hrs 56 min of running time.  This time it was only taking up about 3.7 GB in VRAM and about 11.8 GB total System Memory (including the OS and system processes) so it had not dumped to the CPU.

    So still perplexed as to why it's taking so long as the noise filter was disabled and  in the Progressive tab the Render Quality is only at 2 (which is what I usually use for the a few tests before the final render). 

    Latest test attached.

    There is still a slight bit of noise around the two red lamps on the sides of the door.  

    multikey test 2.jpg
    933 x 1200 - 620K
    Post edited by kyoto kid on
  • FishtalesFishtales Posts: 6,161

    As I said previously. Drop the camera slightly so that the two bright spots of the lights above the door are out of shot and see if that speeds up the render. I have found that any bright spots of light in camera view can slow a render down.

  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ...ah but those didn't adversely impact render times in earlier attempts. If you look at the first image I attached in the OP you will see they are also visible and that took just a little over an hour to complete. I do have an older GPU which is not as fast as an RTX card is running on PCIe 2.0, and this is a fairly dark interior scene which does take a bit longer than a brightly lit outside one.  Just that it shouldn't have taken nearly 3 hours to reach 85%..

    As it's late (almost 00:30 local time) I'll take a look at the render settings again tomorrow.

    Post edited by kyoto kid on
  • kyoto kidkyoto kid Posts: 41,197

    Update.

    Well did some more tests.  Made sure all noise filters were off, there were no other settings that could slow the process, and ran it again however the same situation occurred.  I even ran a test as suggested with the two floodlight emitters out of view of the camera and the same result.  The process appears to slow down at around the 5 min 15 seconds point  with about 520 iterations in and 29.79% of the process complete. This goes on for about a half hour and another 4,200 iterations when it finally reports a convergent percentage again.(29.83%).   

    Now I am at a loss as this didn't happen with just the normal photometric point light and spotlight I original used (and which had the floodlights in clear view of the camera as seen in the first image attached to the OP). So this leaves be back to suspecting the  Horror And Action Lights are the culprits.  Unlike Code 66's Emission Profile Master (which can be applied to any mesh light), these already have lights (Photometric point) with fixed IES profiles attached. 

    I will run another render test with the two HAL lights off and the original point light and spotlight instead.

    Why I turned to these lights is because I wanted a tighter focused beam to more like flashlight which the standard photometric lights don't 

  • marblemarble Posts: 7,500

    Might not be relevant but I have been convinced for a long time that using emissive lights slows down the render considerably. Even Ghost Lights which, I think, are supposed to do the opposite. Whenever possible I use HDRi and a spotlight even now that I have a 3090 (funny how 10 minutes for a render now seems like a long wait although it would have taken over an hour on my old 1070).

  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ...the scene has the same number of emissive mesh lights it had before I swapped out the two standard photometric lights for the ones from the Horror And Action Lights, which are also photometric point lights, (not mesh lights) with preset IES profiles, and it rendered much faster.  An HDRI would defeat the effect I am after as the setting isn't supposed to be brightly or fully illuminated.

     

    Post edited by kyoto kid on
  • FishtalesFishtales Posts: 6,161

    Why not try the IES Profile on your original lights and see if it slows the render down?

  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ...added an IES profile to the floodlights (using the Emission Master Pro script) and no perceivable difference save for the nice effect..

    Seems it's something else as like I mentioned I was going to do another test with the two Horror Action Lights turned off  and the original photometric spot/point lights on (which I just did), however the same situation occurred.  It seems when the total convergence on the main progress monitor reaches 30 - 32% it stops reporting any further progress in convergence.  it still increments render time, iterations, and notes that the process has written to the canvas,  in the scrolling window below, but no further convergence data like I usually would see while the main progress monitor at the top just hangs at between 30 - 32%.  It is acting as if I still have the noise filter active but that is totally shut down.  I expect progress to be slower near the end of the process, but have never seen it happen this early, even when was running earlier tests on this particular scene.   the process isn't dumping from teh GOU (takes up about 3.7 GB out of 12 I have according to Afterburner) though it seems like it has when after nearly 3 hours it's only at about 61%. Went through the diagnostics on my GPU and it checks out as healthy so that isn't the cause.

    The only thing left I can think of anymore is I've hit some sort of bug in the render progress tracking/reporting (this is a beta after all) thoughg I've never seen it happen before) as in the render window it appears to be getting cleaner and cleaner but the progress monitor is still showing that the process is only at about 1/3rd completed..  Makes me feel like the old Reality/Lux days which left determining the progress up to how good your eyes and display were along with some guesswork.

    really baffled with this one.

    Post edited by kyoto kid on
  • marble said:

    Might not be relevant but I have been convinced for a long time that using emissive lights slows down the render considerably. Even Ghost Lights which, I think, are supposed to do the opposite. Whenever possible I use HDRi and a spotlight even now that I have a 3090 (funny how 10 minutes for a render now seems like a long wait although it would have taken over an hour on my old 1070).

    Each polygon of an emissive light adds paths to each iteration of the render process. HDRIs will be fast as they, generally, provide all round light so there are usually relatively few sahdowed nooks; ghost lights are a speed aid for interior renders rather than open air (they may be useful in open air renders for other reasons, however).

  • kyoto kidkyoto kid Posts: 41,197

    ....that is why I didn't use those either as the location is supposed to be somewhat dark. I still don't understand the slowdown so early in the process which is occurring and which didn't present itself in earlier render tests (in which there were more emissive mesh lights).

  • emissive lights definitely add to the render time, I think, I'm pretty sure...
  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ...true, they do.  However I've created other scenes in the past back when I was still rendering in CPU mode which had more emissive lights than this one and many didn't take much longer the the one in question took just to reach 65%.

    Attached below is one of the scenes I rendered that had a fairly large number of emissive surfaces.

    The scene is even darker than the one I am working on. Each of the actual "fireflies" is an emissive as are the candles and the lamps and a ghost light for the "moonlight" effect (19 total emissive items) and it took somewhere around 4 hours on the CPU for the render process to complete. Render size 1,600 x 1,200. 

    The one I am currently working on is not as complex, has less than half that many emissive lights (8) along with one photometric light, is only 933 x 1,200, and rendered on an older but fairly powerful as well as capable GPU taking 3 hours to reach only 65% I would estimate a final render time in the realm of between 5  to 6 hours.  

    rememberance.jpg
    1600 x 1200 - 913K
    Post edited by kyoto kid on
  • kyoto kidkyoto kid Posts: 41,197

    ...OK looked through the logfile on last night's tests and wile a bit of it seems confusing. I found that after Iteration #11, the following warning popping up every so often.

    2021-08-08 00:25:13.104 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported

    The interesting part is that I have the optix-prime accelerator disabled. I would think the warning wouldn't appear if the option was turned off.

    A few lines  later he warning appears after every iteration update.  

    I am using the 451.48 driver. 

  • As far as I remember the Optix prime option doesn't do anything (on account of not being supported) but the check box behaviour was inverted.

    I can't find any patch notes about it other than it being removed completely in 4.12.1.x somewhere.

  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ...so if it's unchecked it's active? 

    I'm still running the 4.12.0.47 Beta so wondering if it is still active in my version  I found this particular release to be so stable even compared to the general release that followed so I just kept using it. 

    I know that the driver version I have is required for 4.15 and read somewhere here about Optix being permanently "activated" for older GPUs. thus using more memory and slowing down the process.  Very reluctant to install 4.15 given all the bugs I keep hearing about.

    If the forum search wasn't a useless peace of rubbish I would look for that reference you mention.

    Post edited by kyoto kid on
  • prixatprixat Posts: 1,590
    You're already using an Iray with permanent optix.
  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ...is it because of the driver?

    I could roll back as I have the older installers.

    Post edited by kyoto kid on
  • I think the version of Iray in 4.12 regardless of drivers already did the automatic optix thing, and the check box was basically legacy as its not something the user sets any more.

    I know finding the problem is the ideal here but as a stop gap measure you could try washing the scene out with a fill light and tonemapping back down.

    Alternatively change the render settings to just use a max time and samples instead of a convergence and set the sample count to say 1000 more than whatever it reached in your tests, then run the render again and see how it turns out, if it stops reporting progress without needing to calculate convergence it may be a bug. Otherwise it could just be that the way the light paths bounce is not great for hitting the darker outlying spots, so the total convergence per sample drops off dramatically to the point where its not enough to report, which is where bumping all your lights up like 2x or 3x and using 32bit light path canvases to merge them all back together might be usefull (depending on your opinion on using canavses) as you can then control the light intensity and tone mapping of each light in post.
     

  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ..the issue with using a "wash light" is that it ruins the effect I am going for with the scene. I am already using 15,000 samples as the maximum (the default) .. Bumping up the lights will make the scene too brightly lit and washes some of the details out.  This is why I think I hit a bug (though this hasn't happened with other even darker scenes I've done using this particular version).

    I am very bad at composting, particularly when shadows fall on different "planes" in the scene which would require having to manually paint them in in a 2D programme as I do not have a deft enough hand to do so.

    Optix is supposed to be an "accelerator", but here it is doing the opposite. Sounds as if Daz broke the programme by forcing Optix on everything rendered. The only option I can see would be going to an older version (4.10 which I still have installed) but the scene will not open that version which means having to rebuild it from scratch.

    I'm going to roll back to an older driver and see what happens as the 451.48 driver is specified for 4.15. .

     

    Post edited by kyoto kid on
  • DustRiderDustRider Posts: 2,785

    To be fair, I don't think DAZ forced optix on at all times, I think it was Nvidia/Iray that did it. If so, then you may need to roll back the driver and/or DS to get control of it again. There is a thread with information about it in the forums, but I had zero luck finding it with a quick search using Google. 

  • kyoto kidkyoto kid Posts: 41,197
    edited August 2021

    ..so did a little digging around after dinner. 

    4.12.0.47  uses Iray version 2019.1.2 build 317500.3714 (carried through from the 4.12.0.42 Beta).

    I found this on the Nvida Release notes for that version:

    • Fixed the issue of the OptiX library not being found on systems with integrated graphics or non-NVIDIA GPUs.

    As I mentioned above, I was getting warnings about that the optix_prime_scene option was no longer being supported. however my system has an Nvida GPU.

    I'm going to roll back to the driver used in 4.10 (Iray 2017.0.1, build 287000.7672) since I still have 4.10 installed as all the 2019 drivers are optimised for RTX GPUs and require 430..86 or higher and if it will open i nteh older version, do some more tests.


    ETA

    OK the scene opened (usually scenes created and saved in a newer version don't).  Seems to go reasonably faster. Reached 50% in a little over an hour and the process. finished at 67%, on 15,000 iterations (used the default samples setting) and took 1hr 28min.

    Before some of the tests I did the other night, I removed the ceiling panel and used the geometry editor to remove most of the set  that didn't affect the scene (as the wall and floor sections were each a single mesh) to see if that had an effect .  It maybe shaved a few minutes off but nothing significant as at the 1h 30m mark and over 12,000 iterations it was still only at best, 26% complete and still rather noisy. 

    This test still has some noise but is far cleaner, so I think it is due to changes in the Iray engine as 4.12 had the first version of Iray that supported RTX cards and I vaguely remember reading something about non RTX cards taking a bit of a performance hit.   Glad I kept 4.10 around (though I have the installer backed up). 

    BTW, none of those Optix warning statements either. 

    Was going to attach the image but forgot to look at the save path, which was set to an old folder that I had since moved to another drive a while back so the file went off into the ozone.  Will render it again using the same settings.


    ...OK here's the image.

    Leela Multikey Daz 4-10 test.jpg
    933 x 1200 - 722K
    Post edited by kyoto kid on
  • kyoto kidkyoto kid Posts: 41,197

    ...not running any tests for the next few days as we are experiening a heatwave with temperatures above 100° and I don't have AC.

Sign In or Register to comment.