Iray memory leak?

Hi all!

I'm trying to render an image series of two frames, both are identical, only the camera changes. The first frame always renders out fine using my GPU (Titan X), so I know it fits on my GPU RAM but for some reason the second frame always falls back on the CPU, even though it is eidentical to the first frame so should fit on my GPU RAM too. I have rendered tons of similar scenes out without any issue, but as of today that changed somehow... Thanks for helping me out with this one...

Kind regards,

Me

Comments

  • OZ-84OZ-84 Posts: 137

    Great, i am not alone! Observed this behaviour since a few days. Checked gpu-z. After finishing the first frame there are about 500-1000mb left in vram memory. Next scene doesnt fit anymore and daz switches to cpu. Thats really damn frustrating!

    it would help if the batch rendere would shit down daz after the render of the first frame, reopen it and render the second frame. What also would be helpfull would be a daz integrated and own batch renderer that flushes vram after finishing a render...

  • Thanks for the reply! But what I don't get is that this worked up untill today. I usually render up to 28 renders like that, one straight after the other... Why doesn't it work anymore? I did not update DAZ Studio or Nvidia Drivers... Yet it somehow fails. I use this method of rendering (using an image series) for the exact reason that Iray would keep all the necessary data in the RAM in between renders so I can render without loading times in between since we're talking about identical scenes. I reduced loading time in between renders to 5 seconds using this method. So why is Iray trying to reload all my data now... It shouldn't, it's already there....

  • OZ-84OZ-84 Posts: 137

    Even if it is reloading ... why the heck isnt it flushing the data? I did a few thousand renders and just a few days before this started to show up. Now i have to modify all scenes i have safed to clear up some vram. Its such a mess. In adition you never know if something stops to work if you let it render thourgh the night. I am so frustrated by this, i would even consider to buy a new GFX card to handle this problem. Point is, i am on 1080tis at the moment and 12gb wouldnt be a real upgrade. Sad that iray isnt willing to put additional data into system ram (like octane is able to do).

    Dont know why this situation is so bad now ... and dont know how get rid of the problem :-( 

  • For me this started to go wrong yesterday... Does someone know how to write a script to make Iray flush the data of the GPU RAM after a render? Or even better, how to force Iray into only loading all resources into RAM before the first render of an image series render batch and cancel out the reloads?

    Thanks a lot,

    Me

  • JamesJABJamesJAB Posts: 1,760

    So, I think your issue comes from leaving that first render windows open.  As long as that window is open, the render is completely loaded and ready to continue.
    For example :
    After a render hits the default 5000 itterations, or 95% convergence it completes.
    You can still go in and change settings on the render and hit continue.

    *** Enter render number two ***

    You move the camera and start the second render.
    Now Iray needs to do a whole new set of lighting calculations in VRAM based on the new camera location.
    The new camera location may also change the Subdivision level of some (or all) of the items in the scene.

    There will always be more VRAM usage on consecutive renders if the previous one(s) are left open.

  • VisuimagVisuimag Posts: 570
    JamesJAB said:

    So, I think your issue comes from leaving that first render windows open.  As long as that window is open, the render is completely loaded and ready to continue.
    For example :
    After a render hits the default 5000 itterations, or 95% convergence it completes.
    You can still go in and change settings on the render and hit continue.

    *** Enter render number two ***

    You move the camera and start the second render.
    Now Iray needs to do a whole new set of lighting calculations in VRAM based on the new camera location.
    The new camera location may also change the Subdivision level of some (or all) of the items in the scene.

    There will always be more VRAM usage on consecutive renders if the previous one(s) are left open.

    Yup! I've had this sort of problem for well over a year now (though I have others that just can't be fixed no matter what). In regards to this issue, I'd alos close down render windwos. What you should consider is using (or creating) the Aux Viewport Tab for Iray Preview and then use the Viewport Tab for the actual renders. This way, the Preview will keep Iray ready to go without having to use all that VRAM to keep previous windows open!

  • sextinosextino Posts: 37
    edited December 2017

    I can't speak for anyone but myself but when I run an image series I don't have a display window for the first render.  I do have the preview window but I never leave it in IRAY mode while rendering.  I have two 1080TI's and a 980TI.  Over the last year Daz has started to use quadrouple the amount of VRAM that it used to.  I used to be able to run scens with 7 Genesis 3 characters in a room and keep in a 6GB envelope.  Now it takes 10Gb just to do 4 genesis 3 characters in the same room. 

    Anyway, My advice is to turn off Opti X if you are using it.  I noticed with the new Nvidia drivers and the latest version of DAZ that Opti X will cause my render footprint to rise 2GB and if you do an image series it will normally clear a couple gigs of VRAM but not if you have Opti X enabled.  My current project consumes 9.8GB of RAM on the first render and when it switches to the second render it doesn't clear anything and it overflows my card.  If I turn off opti X it will only use 7.5 GB of VRAM and then it will rise to 9GB on the seconds render.  After that it will clear the RAM back down to 7.5 GB after each render in the image series so that the VRAM usage never goes over 10 GB. Why it waites until it's finsihed with render two before it starts clearing the VRAM is behond me.  It wasn't like that last year.  

    It's pretty crazy in my opinion that a small room with 3 people in it needs 10GB of VRAM especially when it only consumed 5GB a year ago.  I think either IRAY or DAZ needs some serious optimization.  My 980TI is all but useless at this point.  My 1080TI's can hardly do basic scenes.  You basically have to add 2GB of VRAM to your project if you want to run an image series because of this problem.  If you plan to use Opti X on an image series then you will need 4GB more VRAM than you would otherwise.  2GB because it doesn't clear the memory after the first render and another 2GB for unknown reasons while using Opti X.  If your using Windows 10 then kiss another 2GB of VRAM goodbuy.  I spent $1500 and doubled my VRAM  and yet I am limited far more than I was last year with two 980TI's.  Pretty frustrating.  It's fine if you just want to do one image of a person or two but anything more than that and even the $3000 TITAN card from Nvdia would kick out to the CPU.  Unless we see a 20GB card in the near future even the highest end rigs are going to be toasters.  Things are getting sloppy, If Nvidia, Microsoft and DAZ actually got it togather my average project would not top 6GB.  Even large set pieces with huge textures would fit in an 8GB foot print if things were working correctly.

    Post edited by sextino on
  • OZ-84OZ-84 Posts: 137
    JamesJAB said:

    So, I think your issue comes from leaving that first render windows open.  As long as that window is open, the render is completely loaded and ready to continue.
    For example :
    After a render hits the default 5000 itterations, or 95% convergence it completes.
    You can still go in and change settings on the render and hit continue.

    *** Enter render number two ***

    You move the camera and start the second render.
    Now Iray needs to do a whole new set of lighting calculations in VRAM based on the new camera location.
    The new camera location may also change the Subdivision level of some (or all) of the items in the scene.

    There will always be more VRAM usage on consecutive renders if the previous one(s) are left open.

    James, i am on batch renderer. So, no viewport isnt even open.

  • OZ-84OZ-84 Posts: 137
    sextino said:

    I can't speak for anyone but myself but when I run an image series I don't have a display window for the first render.  I do have the preview window but I never leave it in IRAY mode while rendering.  I have two 1080TI's and a 980TI.  Over the last year Daz has started to use quadrouple the amount of VRAM that it used to.  I used to be able to run scens with 7 Genesis 3 characters in a room and keep in a 6GB envelope.  Now it takes 10Gb just to do 4 genesis 3 characters in the same room. 

    Anyway, My advice is to turn off Opti X if you are using it.  I noticed with the new Nvidia drivers and the latest version of DAZ that Opti X will cause my render footprint to rise 2GB and if you do an image series it will normally clear a couple gigs of VRAM but not if you have Opti X enabled.  My current project consumes 9.8GB of RAM on the first render and when it switches to the second render it doesn't clear anything and it overflows my card.  If I turn off opti X it will only use 7.5 GB of VRAM and then it will rise to 9GB on the seconds render.  After that it will clear the RAM back down to 7.5 GB after each render in the image series so that the VRAM usage never goes over 10 GB. Why it waites until it's finsihed with render two before it starts clearing the VRAM is behond me.  It wasn't like that last year.  

    It's pretty crazy in my opinion that a small room with 3 people in it needs 10GB of VRAM especially when it only consumed 5GB a year ago.  I think either IRAY or DAZ needs some serious optimization.  My 980TI is all but useless at this point.  My 1080TI's can hardly do basic scenes.  You basically have to add 2GB of VRAM to your project if you want to run an image series because of this problem.  If you plan to use Opti X on an image series then you will need 4GB more VRAM than you would otherwise.  2GB because it doesn't clear the memory after the first render and another 2GB for unknown reasons while using Opti X.  If your using Windows 10 then kiss another 2GB of VRAM goodbuy.  I spent $1500 and doubled my VRAM  and yet I am limited far more than I was last year with two 980TI's.  Pretty frustrating.  It's fine if you just want to do one image of a person or two but anything more than that and even the $3000 TITAN card from Nvdia would kick out to the CPU.  Unless we see a 20GB card in the near future even the highest end rigs are going to be toasters.  Things are getting sloppy, If Nvidia, Microsoft and DAZ actually got it togather my average project would not top 6GB.  Even large set pieces with huge textures would fit in an 8GB foot print if things were working correctly.

    Oh funny... you made me check my old renders i did on a single 980ti and you are absolutely right. I have rendered similar scenes on it alone. Now i am wondering how they could fit into the vram and my 1080tis cant handle them now. Sure, assets like hair and and so got bigger. However, if i remember correctly gen8 has less geodata and the textures arent bigge than those used for gen3. I cant get rid of the feeling something is wrong here ... I feel so limited because i can only use 4 or so char models in a scene. If there was the possibility to buy better hardware, and no, i dondt see those 3000$+ "pro" graphic cards as a valid possibility, i would surely upgrade. 

    For now i have started doing some renders without OptiX to see if vram load benefits. 

  • Hi!

    My viewport isn't open either... But I do like the SubD option here. I've noticed that for some reason my figures "jump up" to SubD 2 even though I set them to SubD1 on the first frame. Now obviously with seven G3 figures, if all of them go to SubD2, VRAM can't hold it anymore... I checked all frmes and all figures and set their SubD to 1 and locked that setting, but still Iray starts using the CPU after render number 1. I'll try using with Opti X disabled. 

    On another note: I read something about a Nvidia Volta GPU of 48 GB of RAM... But obviously the way to go here is to have Iray render on GPU while being able to use system RAM like octane does... If there's people here that can write a script to make Iray do toon renders, there must be people here that can write a script to enable Iray to use system RAM and render in GPU mode... There is a similar script for Blender cycles.

    Thanks a lot,

    Me

  • Hi all!!

    Turning off Opti X works for me so far. I'm rendering a series of 16 images and I'm up to render 4 now, all still on GPU. Here's hoping this keeps working! Also I see little difference in end result as far as the number of itterations goes. I end the render after 1300 seconds and it looks like the number of itterations is not at all lower... So far so good...

    Thanks a lot,

    Me

  • WandererWanderer Posts: 957

    I'm not sure what's the deal currently, but I'm ready to throw a brick through the screen--not really. But it is extremely frustrating. Working on a scene, no matter what I do, cannot get it to work in memory. Even tried splitting it into several parts. One part shows 3 Gigs of geometry memory used no matter what I do. Nothing I do will bring the geometry memory used under 3 Gigs for 1 G8F figure with hair, clothing, and a few small props. If I hide everything then it seems to work. Well, that's not helpful, is it? So I decided to try other part of scene. Nothing in scene but a campfire and ground with some emmissives. Nothing else. Go to render, straight to CPU. !??!?! It doesn't even say how much memory is being used by the scene, just starts straight to CPU render. So, I open up GPU-Z while rendering the scene on my CPU only to see what's up with my GPU. It shows that of 6144 MB max, 5123 is used. ?!!?!?? FOR WHAT? No others programs running. Render window is closed after each render. Flushed video memory with script several times. Tried creating new scene, restarting Daz, all the usual tricks. WTWTW!?!?! Daz, where did my VRAM go? 

  • dragotxdragotx Posts: 1,138
    How long has it been since you restarted Studio? There is a definite memory leak in the program, especially when you've been rendering in Iray. It essentially doesn't let go of all of the vram (or system ram either to be honest). Small scenes you can get away with rendering several sequentially, but a big one should have studio bounced between each one. And if it ever drops down to CPU, it won't go back to GPU until it has been bounced.
  • WandererWanderer Posts: 957

    Looking at GPU-Z when I closed Daz, and the VRAM on my card immediately dropped to 244 MB. Hmm.... ya think? Thanks @dragotx... my brain hurts less.

  • WandererWanderer Posts: 957

    I hate to say this, because I really like Daz Studio, but if things continue as they are, I may need to find another solution for rendering. Every large render I do locks up between 1-1/2 to 5 Gigs of VRAM. It's so bad that I often have to save a scene, close Daz, restart Daz, load scene, render, make changes, save, exit daz, restart daz, ad nauseum. I don't know what the deal is, but surely it can be fixed. Please fix it, guys. Just saying.

  • Yah this issue should have been fixed some time ago. I generally save and exit daz before rendering just to make sure that all my recources arent tide up.

  • fastbike1fastbike1 Posts: 4,078

    @Wanderer "Every large render I do locks up between 1-1/2 to 5 Gigs of VRAM."

    What is it you expect? IME those aren't unexpected values for large or complex renders without any optmization. That Vram is released as soon as the render window(s) are closed. My installation of Studio will drop back to whatever the pre-render value was after all render windows are closed.

  • FishtalesFishtales Posts: 6,162

    I don't have a graphics card but some of my renders can push the 16 GB of ram I have installed on my laptop to the limit and beyond so it is into the paging file. I have a program installed which I can click on to clean ram and I can get anything up to 2 GB back. Whether that works with VRAM I have no idea.

  • I second that issue guys. Been rendering on an 8 x GTX 1070 Iray server and suddenly, DAZ Studio switches to using the CPU... Kind of ruins the whole thing, since it's 15-30 times slower... lol I'm rendering to files as images series... 

  • kyoto kidkyoto kid Posts: 41,222

    ...what is worse is when the process dumps to swap mode as I used to experience regularly before upgrading the memory on my system.  At the time I was still stuck with Iray CPU rendering, and having the scene itself open in Daz alone would take up a major part of the 11 GB I had (I say 11 GB as that is pretty much what the useable memory is after Windows and system utilities).

  • edited June 2019

    Disabling OptiX Prime Acceleration solved the issue for me of DAZ switching to CPU from GPU rendering after rendering the first image in image series. Running GTX 1070 with 419.17 driver (DAZ 4.10.0.123 Pro 64-bit)

    Post edited by dennis.d.t@outlook.com on
  • IvyIvy Posts: 7,165
    edited June 2019

    What version you guys using to get these vram memory leaks issues?

    I've been building huge rendered scenes . with 2- 1080ti & I've been barely using 8 gigs loaded on 1 card. . I have noticed my system rams usage has more than doubled. but not the vram. & i contributedt my ram issues  to some bottle necking around my CPU 

    I'm still using 4.10.0.123 as my main software.  I have the daz 11 beta.  but i don't use beta versions for commissioned renders.

    That is why i asked which version of daz your using. and also have you recently updated your gpu drivers the other day I got a NIVIDA  drive update that i had to roll back because it kept crashing daz 10 could your leak issues be due to a driver update?

     

    Post edited by Ivy on
  • CyberdeneCyberdene Posts: 72
    edited November 2019
    OZ-84 said:

    Even if it is reloading ... why the heck isnt it flushing the data? I did a few thousand renders and just a few days before this started to show up. Now i have to modify all scenes i have safed to clear up some vram. Its such a mess. In adition you never know if something stops to work if you let it render thourgh the night. I am so frustrated by this, i would even consider to buy a new GFX card to handle this problem. Point is, i am on 1080tis at the moment and 12gb wouldnt be a real upgrade. Sad that iray isnt willing to put additional data into system ram (like octane is able to do).

    Dont know why this situation is so bad now ... and dont know how get rid of the problem :-( 

    I strongly agree. If I could I would give up and look for a different method to render. There sadly are no other programs I can switch where I can use all my Iray content. I don't think Nvidia gives a goddamn, and I've fussed with DAZ over this issue as well. I sometimes feel like iRAY is outdated with this whole memory issue thing. My machine cost a lot, and I have over 16GB, yet I still can't render complex scenes like I really want. IRAY is literately a Portrait renderer which means if your using it to make large scenes with it, it will be a sad disappointment...Probably why I'm not that impressed by half the renders I see because nobody is doing anything complex with IRAY, just a ton of portrait renders. Everyone I know that uses IRAY tells me the memory issue is whats holding them back. So whenever I do see complex renderers, they definately aren't done with the IRAY renderer but the artist is using DAZ content so most likely they are using something else to render their DAZ content with. I hate limitations and restrictions because I dont like portrait renders, I like making complex scenes because I enjoy making comics, telling stories, etc portraits cant really do that....Wish Nvidia would just bust out with a 1000GB VRAM card. Now that would be true progress.

    Post edited by Cyberdene on
  • joseftjoseft Posts: 310
    edited November 2019
    Cyberdene said:
    OZ-84 said:

    Even if it is reloading ... why the heck isnt it flushing the data? I did a few thousand renders and just a few days before this started to show up. Now i have to modify all scenes i have safed to clear up some vram. Its such a mess. In adition you never know if something stops to work if you let it render thourgh the night. I am so frustrated by this, i would even consider to buy a new GFX card to handle this problem. Point is, i am on 1080tis at the moment and 12gb wouldnt be a real upgrade. Sad that iray isnt willing to put additional data into system ram (like octane is able to do).

    Dont know why this situation is so bad now ... and dont know how get rid of the problem :-( 

    I strongly agree. If I could I would give up and look for a different method to render. There sadly are no other programs I can switch where I can use all my Iray content. I don't think Nvidia gives a goddamn, and I've fussed with DAZ over this issue as well. I sometimes feel like iRAY is outdated with this whole memory issue thing. My machine cost a lot, and I have over 16GB, yet I still can't render complex scenes like I really want. IRAY is literately a Portrait renderer which means if your using it to make large scenes with it, it will be a sad disappointment...Probably why I'm not that impressed by half the renders I see because nobody is doing anything complex with IRAY, just a ton of portrait renders. Everyone I know that uses IRAY tells me the memory issue is whats holding them back. So whenever I do see complex renderers, they definately aren't done with the IRAY renderer but the artist is using DAZ content so most likely they are using something else to render their DAZ content with. I hate limitations and restrictions because I dont like portrait renders, I like making complex scenes because I enjoy making comics, telling stories, etc portraits cant really do that....Wish Nvidia would just bust out with a 1000GB VRAM card. Now that would be true progress.

    Nvidia were saying they were 'looking in to' out-of-core functions similar to octane and other renderers, as far back as 2015. 

    I think a big reason why it still isnt implemented into iRay is because iRay can run with CPU power if the scene data exceeds the GPU VRAM. 

    Prior to Octane implementing out-of-core functionality, if the scene did not fit into the GPU VRAM, it would not render. You would have to try and optimise your scene or find workarounds like rendering scenes in parts and then compositing in photoshop etc. 

    So Nvidia probably treat this as a low priority, because iRay will still work if you dont have enough VRAM, just slower than it would otherwise. 

    Im sure there are other reasons too. But i would not be suprised if they start to put more resources into iRay development, given the recent advancements in GPU rendering and GPU's themselves, eventually CPU rendering will be redundant, if its not already. 

    I know there are major studios that still use CPU power though.

     

    Post edited by joseft on
  • kyoto kidkyoto kid Posts: 41,222

    ...it would have been a good move if they did. Iray on the CPU is about as fast as a slug travelling through molasses, though not quite as glacial as old Luxrender was.  Of course it could also be a marketing thing to get people to purchase higher VRAM (more expensive) GPUs as out of core rendering (the way Otoy approaches it) is still much quicker than pure CPU rendering. 

    I've been considering going the Octane4 subscription route (the 20$/month one which also includes the Daz plugin).

  • marblemarble Posts: 7,500
    kyoto kid said:

    ...it would have been a good move if they did. Iray on the CPU is about as fast as a slug travelling through molasses, though not quite as glacial as old Luxrender was.  Of course it could also be a marketing thing to get people to purchase higher VRAM (more expensive) GPUs as out of core rendering (the way Otoy approaches it) is still much quicker than pure CPU rendering. 

    I've been considering going the Octane4 subscription route (the 20$/month one which also includes the Daz plugin).

    At least Luxrender could run on two PCs at the same time over the network. I used to do this with an iMac and a Linux PC networked together and could get decent renders in less than an hour. Also, Luxrender runs in the backgorund so you can be working on your next scene in DAZ Studio while the previous one is rendering. Nevertheless, GPU is much faster but this problem with it falling back to CPU on the second frame of an image series is why I reverted to 4.11 which still allows Optix Prime to be switched off. 

    I don't see why I should shell out $$$ for Octane when I've already spent $$$ on a GPU to render in IRay. 

  • kyoto kidkyoto kid Posts: 41,222

    ...well, it depends on just how much VRAM that GPU has.  If it's only 6 GB, Octane could be a good choice to avoid having to do a lot of optimisation/postwork. If you have a 12 GB Titan with (or a RTX model with 24) then yeah, not much to worry about.

    Working with Luxrender, I did like the fact it ran in background and you could also pause, shut the system down, start it up the next day and resume. Speed was really dependent though on having as many CPU cores as you could afford (which back then was more expensive than a GPU card).  Yeah, that was when I started toying around with the idea of dual socket workstation/server MBs 

    I tried using the speed boost but only got about a x2 - x3 gain (rather than the advertised x10) with my older CPU and render quality definitely suffered.  Reality 4.just was too buggy and patches released for one issue tended to break something else, so I stopped using it.

    Yeah the perpetual licence of Octane is more than I can afford but the subscription track makes it a lot easier for those like myself a tight budget.

  • This issue still exists and is a program bug. Even with a scene only using 1gb and using a more than capable GPU, it fails.
  • hobhobuk said:

    This issue still exists and is a program bug. Even with a scene only using 1gb and using a more than capable GPU, it fails.

    Using 1GB for what? The amount of memory a scene takes while being worked on in DS does not necessarily relate to the amount of memory it uses in Iray as the two use different sub-sets of the data with different optimisations.

  • AllenArtAllenArt Posts: 7,172

    hobhobuk said:

    This issue still exists and is a program bug. Even with a scene only using 1gb and using a more than capable GPU, it fails.

    Do you have a dedicated render card or are you using the same card that's also running your monitors?

Sign In or Register to comment.