Flush GPU Memory between Renders
Gazukull
Posts: 96
Hey Ya'll,
So let me explain this behaviour. I will put like 40 images into the batch renderer and let it go over night. Sometimes, after 12 or so, Iray no longer uses the GPU's and switches to CPU mode (thus the images don't clear up in the alloted time).
The only thing I can think, is that there is a memory leak, or the GPU memory is not flushing properly between renders (or something else entirely that had not occured to me).
So... should I report a bug and see if the DS programmers have a solution or is this a better forum to seek a solution (like a script you can run to clean GPU memory).
Thanks to anyone who reads this.
-G
Comments
Since you are asking about GPU, I'm assuming you are rendering in Iray, correct?
Are you rendering directly to a file or to a new window?
If you are rendering to a new window, the behavior is intentional. Some amount will be held as long as the render window remains open. This is to allow resuming the render.
Rendering directly to a file should not hold anything once the render is complete.
Hi mjc1016!
Yes, using Iray.
The batch renderer renders directly to file. I am using the batch renderer from D3D available on Renderosity. I actually thought there was a similiar product here but I cannot seem to locate it in the store.
All that aside, it really just seems like the GPU memory is not being emptied between files. My work around so far is, reboot system load the batch up and let it run until the GPU's stop working on the files (usually after 12 or so).
It's a bummer.
There's a script available somewhere, that allows you to specify the frame numbers you want rendering. Think I saw it on the free section here.
I also recall seeing a flush memory option somewhere - but unfortuneately can't remember where.
Install GPUz, and keep your eye on it for a while, it will let you know how much memory is being used.
Also Check if one of your renders was to big for your GPU.
When GPUs are failing due to out of memory they don't allways recover for the next render without restarting DS.
Take a look at this thread.
http://www.daz3d.com/forums/discussion/44350/scripts-delete-keyframe-figure-and-lock-parts-body-for-ds3-and-ds4-update-v1-3#latest
There is also situations where it is helpful to keep the scene information in memory between renders.
Render at a small resolution; stop as soon as the image appears in the render window.
Minimize the render.
Now anytime you need to use the spot render, it will start rendering much quicker.
In addition, a full render will also start much quicker as many/all of the details are already present on the GPU.
A small render will take up very little GPU RAM, and using GPUz you can confirm it by watching how much it chages by when you close the render window.
I am also having a similar problem. I've addressed it here before but no one really had an answer. I am rendering scenes for a comicbook. Same figures, same environments, same lighting, same everthing but comera angles. After a few renders at times under 20 minutes, the time starts taking close to an hour and then I get kicked to CPU (which is forever!). I have to restart DAZ to get back to normal times, but even sometimes that doesn't work. I can usually get about 8 renders before my GPU looses it mind. I never used to have these issues with 3 delight and miss those old days. But Iray is just so awesome, I wanna figure this out!
Using the latest down load of Daz 4.9, Nvidia 1060 (3gb), Windows 10
There's a Purge Memory script under Utilities->Scripts if that's what you are referring to...
g.
That seems like it could be very helpful. However, I don't have a Purge memory under scripts in DAZ. Content Library>Scripts>Utilities Is that where it should be? All I have there is "Move Scripts" and "Scene Builder". Is it something I need to down load? Thanks
It used to be part of the main Daz Studio application package; maybe it's part of Default Resources for DS 4.9?
The memory purge script does nothing for VRAM, it's supposed to flush RAM(don't know if it works for RAM, but have tested it for VRAM flush). Closing studio is the only thing that will purge my VRAM back down to the baseline ~500mb used for the OS. Once i render, there is about 1.5G VRAM that is held forever on top of my OS used until I close studio.
I would also take a look at temperatures of your GPU if it is overheating it may be getting too hot. Just install gpuz and check temps after 4 or 5 similtanious renders. Iray is really hard on gpu's. Heats them up!
Yep. This is either a bug in Iray itself (it isn't releasing some of the memory on exit) or in DS (it isn't TELLING Iray to release that memory.) No telling when it will get fixed.
Yup. That's it. No idea how effective it is, as I can't recall the last time I used it.
I use a 980ti for rendering, and the only time that the GPU holds 'stuff' in memory is if I leave a render open to facilitate loading times (which are better with 4.9); otherwise, it is good at releaseing memory.
... The only caveat, I'm trying windows 10 and haven't done much rendering since I installed. I'll check out what happens when I do.
I use a Nvidia 1060 3BG on Windows 10. Daz 4.9 and I can usually get about 6 renders with two G3's, clothing and old interior models that I installed default Iray shaders to. First few I can get done in under 20 minutes, then they start taking over thirty for a couple and then in happens... CPU processing scene. Same models, lights and camera. And I am NOT leaving render windows open. The purge memory script doesn't seem to have an effect.
The VRAM objects are connected to the OpenGL context, so if someone is not freeing some memory it will stay there until the OpenGL context is released (unless there is a bug in the nVIDIA driver of course), but if there is a bug it can be anywhere, it can be in Daz Studio, it can be in IRAY.
Wow! I didn't realize this thread resurfaced. What I ended up doing was buying two TITAN X (Maxwell) off ebay. Worked great!
Well I guess that's one way to solve the issue Gazukull. I'd sure like to find a way to end this problem for me and my 1060 though!
I also observed that Iray seems to choose CPU mode even if only a fraction of the VRAM is used.
Example:
- - -
There are 11 GB of VRAM on a GTX 1080 Ti and for some reason CPU mode is activated with only 1.8 GB of geometry memory used. All other applications are closed. There is no chance that texture memory consumption uses 8GB + on the tested scene.
- - -
Without advanced diagnostic tools how DAZ Studio Iray is using the VRAM we are left to:
Speculation:
My guess is that the Iray "frame buffer" is not working as intended.
The frame buffer is the part of the VRAM that holds the information needed to display the image on the active rendering or live preview viewports.
The image that is currently being rendered is also kept in VRAM.
The larger the image resolution the larger the VRAM needed for the frame buffer.
-> What may be going wrong:
- Data of images allready rendered in the separate render window may be kept in memory even after the image was finished, saved and the window closed.
- In addition there may be unwanted interference between the data of an active Iray live preview viewport and a separate dedicated "final" render window.
- When the user switches between DAZ Studio activites the Iray live preview viewport is reloaded. It is unclear if all the data of previous Iray live preview viewport frame buffers is completly cleared from VRAM or not.
- - -
Commentary:
It is not an accepable solution that customers have to close and restart a DAZ Studio scene just to flush the frame buffer.
It is a waste of time to reload all scene data once again if only the small VRAM section of the "frame buffer" would need to be flushed.
-> Please provide tools that acurrately show how the VRAM is used.
-> Please provide tools to manually control which part of the VRAM data is kept and which part of the VRAM is flushed.
-> Please forward customer feedback to Nvidia Iray developers.
If DAZ3D staff is not going to do that please say so.
-> Then DAZ 3D customers may need to start posting directly on the Nvidia Iray forums and ask for solutions there.
- - -
Odd. Your log snippet doesn't show 'texture memory consumption'. For me, this shows right after geometry memory consumption. Curious how big that number is.
As for 'data of images kept in memory' - I've only run into this when nearing border of my 8GB vram limit and rendering multiple images in sequence (e.g., animation). The fix for me, as mentioned in other threads, is to disable OptiX. Normally I live OptiX on, unless I hit this issue.
As for 'active iray preview' - I never leave iray preview active in main or aux viewport when rendering. I figure: why potentially overburden system. It's not like I'm going to be messing around in Daz much during a synchronous render anyway. Perhaps this is just me knocking on wood, but I always switch back to texture shaded prior to a render out of habit.
All this is still new to me, how can I reduce texture size?
Petra
Easiest way is with this: https://www.daz3d.com/scene-optimizer
Alternatively, there are free texture reducing scripts in the freebies forum: https://www.daz3d.com/forums/discussion/137161/reduce-texture-sizes-easily-with-this-script#latest
This tool has just been updated by Nvidia to include a way to flush or flash the bios and clean up any garbage that could slow or brick a graphics card.
https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exelocal
Also, ther clear memory script in daz3d, I been told, only works with undo history.