Flush GPU Memory between Renders

Hey Ya'll,

So let me explain this behaviour.  I will put like 40 images into the batch renderer and let it go over night.  Sometimes, after 12 or so, Iray no longer uses the GPU's and switches to CPU mode (thus the images don't clear up in the alloted time).

The only thing I can think, is that there is a memory leak, or the GPU memory is not flushing properly between renders (or something else entirely that had not occured to me).  

So... should I report a bug and see if the DS programmers have a solution or is this a better forum to seek a solution (like a script you can run to clean GPU memory).

Thanks to anyone who reads this.

-G

Comments

  • mjc1016mjc1016 Posts: 15,001

    Since you are asking about GPU, I'm assuming you are rendering in Iray, correct?

    Are you rendering directly to a file or to a new window?

    If you are rendering to a new window, the behavior is intentional.  Some amount will be held as long as the render window remains open.  This is to allow resuming the render.

    Rendering directly to a file should not hold anything once the render is complete.

  • GazukullGazukull Posts: 96

    Hi mjc1016!

    Yes, using Iray.

    The batch renderer renders directly to file.  I am using the batch renderer from D3D available on Renderosity.  I actually thought there was a similiar product here but I cannot seem to locate it in the store.

    All that aside, it really just seems like the GPU memory is not being emptied between files.  My work around so far is, reboot system load the batch up and let it run until the GPU's stop working on the files (usually after 12 or so).

    It's a bummer.

  • nicsttnicstt Posts: 11,715

    There's a script available somewhere, that allows you to specify the frame numbers you want rendering. Think  I saw it on the free section here.

    I also recall seeing a flush memory option somewhere - but unfortuneately can't remember where.

    Install GPUz, and keep your eye on it for a while, it will let you know how much memory is being used.

  • RenomistaRenomista Posts: 921

    Also Check if one of your renders was to big for your GPU.

    When GPUs are failing due to out of memory they don't allways recover for the next render without restarting DS.

  • nicsttnicstt Posts: 11,715

    There is also situations where it is helpful to keep the scene information in memory between renders.

    Render at a small resolution; stop as soon as the image appears in the render window.

    Minimize the render.

    Now anytime you need to use the spot render, it will start rendering much quicker.

    In addition, a full render will also start much quicker as many/all of the details are already present on the GPU.

    A small render will take up very little GPU RAM, and using GPUz you can confirm it by watching how much it chages by when you close the render window.

  • sarge74sarge74 Posts: 121

    I am also having a similar problem. I've addressed it here before but no one really had an answer. I am rendering scenes for a comicbook. Same figures, same environments, same lighting, same everthing but comera angles. After a few renders at times under 20 minutes, the time starts taking close to an hour and then I get kicked to CPU (which is forever!). I have to restart DAZ to get back to normal times, but even sometimes that doesn't work. I can usually get about 8 renders before my GPU looses it mind. I never used to have these issues with 3 delight and miss those old days. But Iray is just so awesome, I wanna figure this out!

    Using the latest down load of Daz 4.9, Nvidia 1060 (3gb), Windows 10

  • nicstt said:
    I also recall seeing a flush memory option somewhere - but unfortuneately can't remember where.

     

    There's a Purge Memory script under Utilities->Scripts if that's what you are referring to...

    g.

  • sarge74sarge74 Posts: 121
    nicstt said:
    I also recall seeing a flush memory option somewhere - but unfortuneately can't remember where.

     

    There's a Purge Memory script under Utilities->Scripts if that's what you are referring to...

    g.

    That seems like it could be very helpful. However, I don't have a Purge memory under scripts in DAZ. Content Library>Scripts>Utilities Is that where it should be? All I have there is "Move Scripts" and "Scene Builder". Is it something I need to down load? Thanks

  • fixmypcmikefixmypcmike Posts: 19,601

    It used to be part of the main Daz Studio application package; maybe it's part of Default Resources for DS 4.9?

  • TheKDTheKD Posts: 2,696
    edited April 2017

    The memory purge script does nothing for VRAM, it's supposed to flush RAM(don't know if it works for RAM, but have tested it for VRAM flush). Closing studio is the only thing that will purge my VRAM back down to the baseline ~500mb used for the OS. Once i render, there is about 1.5G VRAM that is held forever on top of my OS used until I close studio.

    Post edited by TheKD on
  • Silver DolphinSilver Dolphin Posts: 1,615
    sarge74 said:

    I am also having a similar problem. I've addressed it here before but no one really had an answer. I am rendering scenes for a comicbook. Same figures, same environments, same lighting, same everthing but comera angles. After a few renders at times under 20 minutes, the time starts taking close to an hour and then I get kicked to CPU (which is forever!). I have to restart DAZ to get back to normal times, but even sometimes that doesn't work. I can usually get about 8 renders before my GPU looses it mind. I never used to have these issues with 3 delight and miss those old days. But Iray is just so awesome, I wanna figure this out!

    Using the latest down load of Daz 4.9, Nvidia 1060 (3gb), Windows 10

    I would also take a look at temperatures of your GPU if it is overheating it may be getting too hot. Just install gpuz and check temps after 4 or 5 similtanious renders. Iray is really hard on gpu's. Heats them up!

  • hphoenixhphoenix Posts: 1,335
    TheKD said:

    The memory purge script does nothing for VRAM, it's supposed to flush RAM(don't know if it works for RAM, but have tested it for VRAM flush). Closing studio is the only thing that will purge my VRAM back down to the baseline ~500mb used for the OS. Once i render, there is about 1.5G VRAM that is held forever on top of my OS used until I close studio.

    Yep.  This is either a bug in Iray itself (it isn't releasing some of the memory on exit) or in DS (it isn't TELLING Iray to release that memory.)  No telling when it will get fixed.

     

  • nicsttnicstt Posts: 11,715
    edited April 2017
    nicstt said:
    I also recall seeing a flush memory option somewhere - but unfortuneately can't remember where.

     

    There's a Purge Memory script under Utilities->Scripts if that's what you are referring to...

    g.

    Yup. That's it. No idea how effective it is, as I can't recall the last time I used it.

     

    I use a 980ti for rendering, and the only time that the GPU holds 'stuff' in memory is if I leave a render open to facilitate loading times (which are better with 4.9); otherwise, it is good at releaseing memory.

    ... The only caveat, I'm trying windows 10 and haven't done much rendering since I installed. I'll check out what happens when I do.

    Post edited by nicstt on
  • sarge74sarge74 Posts: 121
    edited April 2017
    nicstt said:
    nicstt said:
    I also recall seeing a flush memory option somewhere - but unfortuneately can't remember where.

     

    There's a Purge Memory script under Utilities->Scripts if that's what you are referring to...

    g.

    Yup. That's it. No idea how effective it is, as I can't recall the last time I used it.

     

    I use a 980ti for rendering, and the only time that the GPU holds 'stuff' in memory is if I leave a render open to facilitate loading times (which are better with 4.9); otherwise, it is good at releaseing memory.

    ... The only caveat, I'm trying windows 10 and haven't done much rendering since I installed. I'll check out what happens when I do.

    I use a Nvidia 1060 3BG on Windows 10. Daz 4.9 and I can usually get about 6 renders with two G3's, clothing and old interior models that I installed default Iray shaders to. First few I can get done in under 20 minutes, then they start taking over thirty for a couple and then in happens... CPU processing scene. Same models, lights and camera. And I am NOT leaving render windows open. The purge memory script doesn't seem to have an effect.

    Post edited by sarge74 on
  • The VRAM objects are connected to the OpenGL context, so if someone is not freeing some memory it will stay there until the OpenGL context is released (unless there is a bug in the nVIDIA driver of course), but if there is a bug it can be anywhere, it can be in Daz Studio, it can be in IRAY.

     

  • GazukullGazukull Posts: 96

    Wow!  I didn't realize this thread resurfaced.  What I ended up doing was buying two TITAN X (Maxwell) off ebay.  Worked great!

  • sarge74sarge74 Posts: 121

    Well I guess that's one way to solve the issue Gazukull. I'd sure like to find a way to end this problem for me and my 1060 though!

  • linvanchenelinvanchene Posts: 1,382
    edited May 2017

    I also observed that Iray seems to choose CPU mode even if only a fraction of the VRAM is used.

    Example:

     

    Rendering in NVIDIA Iray

    Compiling Shaders - 0/57

    Rendering image

    Rendering...

    Iray VERBOSE - module:category(IRAY:RENDER): 1.13 IRAY rend progr: CUDA device 0 (GeForce GTX 1080 Ti): Processing scene...

    Iray VERBOSE - module:category(IRAY:RENDER): 1.12 IRAY rend progr: CUDA device 2 (GeForce GTX 1080 Ti): Processing scene...

    Iray VERBOSE - module:category(IRAY:RENDER): 1.2 IRAY rend stat : Geometry memory consumption: 1.89079 GiB (device 0), 0 B (host)

    Iray VERBOSE - module:category(IRAY:RENDER): 1.3 IRAY rend stat : Geometry memory consumption: 1.89079 GiB (device 2), 0 B (host)

    Iray VERBOSE - module:category(IRAY:RENDER): 1.13 IRAY rend progr: CPU: Processing scene...

    - - -

    There are 11 GB of VRAM on a GTX 1080 Ti and for some reason CPU mode is activated with only 1.8 GB of geometry memory used. All other applications are closed. There is no chance that texture memory consumption uses 8GB + on the tested scene.

    - - -

     

    Without advanced diagnostic tools how DAZ Studio Iray is using the VRAM we are left to:

    Speculation:

    My guess is that the Iray "frame buffer" is not working as intended.

    The frame buffer is the part of the VRAM that holds the information needed to display the image on the active rendering or live preview viewports.

    The image that is currently being rendered is also kept in VRAM.

    The larger the image resolution the larger the VRAM needed for the frame buffer.

     

    -> What may be going wrong:

    - Data of images allready rendered in the separate render window  may be kept in memory even after the image was finished, saved and the window closed.

    - In addition there may be unwanted interference between the data of an active Iray live preview viewport and a separate dedicated "final" render window.

    - When the user switches between DAZ Studio activites the Iray live preview viewport is reloaded. It is unclear if all the data of previous Iray live preview viewport frame buffers is completly cleared from VRAM or not.

    - - -

    Commentary:

    It is not an accepable solution that customers have to close and restart a DAZ Studio scene just to flush the frame buffer.

    It is a waste of time to reload all scene data once again if only the small VRAM section of the "frame buffer" would need to be flushed.

    -> Please provide tools that acurrately show how the VRAM is used.

    -> Please provide tools to manually control which part of the VRAM data is kept and which part of the VRAM is flushed.

    -> Please forward customer feedback to Nvidia Iray developers.

    If DAZ3D staff is not going to do that please say so.

    -> Then DAZ 3D customers may need to start posting directly on the Nvidia Iray forums and ask for solutions there.

    - - -

    Post edited by linvanchene on
  • TooncesToonces Posts: 919

    Odd. Your log snippet doesn't show 'texture memory consumption'. For me, this shows right after geometry memory consumption. Curious how big that number is.

    As for 'data of images kept in memory' - I've only run into this when nearing border of my 8GB vram limit and rendering multiple images in sequence (e.g., animation). The fix for me, as mentioned in other threads, is to disable OptiX. Normally I live OptiX on, unless I hit this issue.

    As for 'active iray preview' - I never leave iray preview active in main or aux viewport when rendering. I figure: why potentially overburden system. It's not like I'm going to be messing around in Daz much during a synchronous render anyway. Perhaps this is just me knocking on wood, but I always switch back to texture shaded prior to a render out of habit.

  • PetraPetra Posts: 1,156

    All this is still new to me, how can I reduce texture size? 

    Petra

  • TooncesToonces Posts: 919

    Easiest way is with this: https://www.daz3d.com/scene-optimizer

    Alternatively, there are free texture reducing scripts in the freebies forum: https://www.daz3d.com/forums/discussion/137161/reduce-texture-sizes-easily-with-this-script#latest

  • MinamMinam Posts: 55
    edited December 2018

    This tool has just been updated by Nvidia to include a way to flush or flash the bios and clean up any garbage that could slow or brick a graphics card. 
    https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exelocal

    Also, ther clear memory script in daz3d, I been told, only works with undo history.  

    Post edited by Minam on
Sign In or Register to comment.