Loading...
> >

IRAY VRAM Fail-over to CPU

marblemarble Posts: 7,500
in Daz Studio Discussion

I have a scene which I monitor with GPU-Z to keep an eye on the VRAM usage. I have GPU-Z set to record the Highest amount (i.e. the peak usage) and it does not go above 3.3 GB. However, the scene is failing over to CPU. This is the log:

Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Initializing local rendering.Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering with 1 device(s):Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : 	CUDA device 0 (GeForce GTX 970)Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering...Iray INFO - module:category(IRAY:RENDER):   1.9   IRAY   rend info : Initializing light hierarchy.Iray INFO - module:category(IRAY:RENDER):   1.3   IRAY   rend info : CUDA device 0 (GeForce GTX 970): Scene processed in 129.022sIray INFO - module:category(IRAY:RENDER):   1.3   IRAY   rend info : CUDA device 0 (GeForce GTX 970): Allocated 89 MB for frame bufferIray INFO - module:category(IRAY:RENDER):   1.3   IRAY   rend info : CUDA device 0 (GeForce GTX 970): Allocated 804 MB of work space (1024k active samples in 0.259s)WARNING: dzneuraymgr.cpp(261): Iray ERROR - module:category(IRAY:RENDER):   1.9   IRAY   rend error: CUDA device 0 (GeForce GTX 970): out of memory (while launching CUDA ior stack renderer in core_cuda_host.cpp:215)WARNING: dzneuraymgr.cpp(261): Iray ERROR - module:category(IRAY:RENDER):   1.3   IRAY   rend error: CUDA device 0 (GeForce GTX 970): Failed to update ior stackWARNING: dzneuraymgr.cpp(261): Iray ERROR - module:category(IRAY:RENDER):   1.3   IRAY   rend error: CUDA device 0 (GeForce GTX 970): Device failed while renderingWARNING: dzneuraymgr.cpp(261): Iray ERROR - module:category(IRAY:RENDER):   1.3   IRAY   rend error: All workers failed: aborting renderWARNING: dzneuraymgr.cpp(261): Iray WARNING - module:category(IRAY:RENDER):   1.3   IRAY   rend warn : All available GPUs failed.Iray INFO - module:category(IRAY:RENDER):   1.3   IRAY   rend info : Falling back to CPU rendering.Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Retry rendering with other devices.Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : CPU: using 8 cores for renderingIray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : Rendering with 1 device(s):Iray INFO - module:category(IRAY:RENDER):   1.0   IRAY   rend info : 	CPU

The odd thing is that I set up another scene with 3 people, clothed, in a complex interior and GPU-Z reported 3.6GB but it did not fail over.

I have recreated the failing scene from scratch but can't get it to render in GPU mode.

Comments

  • SimonJMSimonJM Posts: 5,997

    No idea, but my guess would be the next VRAM memory allocation request would exceed free VRAM which is why it fails and why you don't see the highwater mark go up.

  • Kevin SandersonKevin Sanderson Posts: 1,643

    I can't recall if you have a separate card for your monitor. If you don't, it's the OS using part of the memory of your GPU along with Iray.

  • marblemarble Posts: 7,500

    I run the monitor from the Intel graphics on the i7 (Skylake) so I should have all the GPU VRAM available for IRay.

    Now I've tried the same scene with half of the content taken out and it is still refusing to render. Another scene with more content is working ok. Confused, I am.

  • mjc1016mjc1016 Posts: 15,001

    Try and isolate which item/items are the problem...because it could be something like an odd texture map that is 'inflating' to an enormous size.

  • marblemarble Posts: 7,500

    I suspect you might be right (which is why I started by removing half the content). But there's another oddity - the same scene with the same content but different poses rendered fine previously. So I'm going back through the iterations of that scene to see if I can figure out when it went wrong. I've closed and rebooted several times.

    The other possibility which I don't want to think about is that the GTX970 might have a fault. 

  • marblemarble Posts: 7,500
    edited May 2016

    I finally called the service dept. at the computer supplier and told them I think there is a problem with the GPU. I had been running memtestG80 which failed with errors. The techie said he'd never heard of that diagnostic and sent me a link to another called FurMark. I'm not at all sure that FurMark actually tests all of the memory and my problem seems to hit somewhere around the 3.2 GB usage. He wasn't at all familiar with rendering outside of games.

    His colleague suggested another memory tester : Video Memory Stress Test but it looks very old (2008) and I'm not sure it works with a GTX970 and recent drivers. They don't guarantee Cuda test results either.

     

    memtest.png
    1103 x 833 - 62K
    VMT1.jpg
    799 x 610 - 72K
    VMT2.png
    794 x 602 - 105K
    Post edited by marble on
  • ToborTobor Posts: 2,300
    edited May 2016

    There are numerous thread on the Web regarding the 970 offering only 3.5G of VRAM for some applications, even though it contains 4G. I'm not sure how, or if, this also applies to Iray. nVidia doesn't always come out and mention this in tech support responses, and there is some disagreement about what's actually happening in the card, but you can research to see if the issue applies to you.

    While GPU-Z is pretty handy, I don't know if it tells you the actual RAM usage, depending on driver, etc. These cards do a lot of on-the-fly optimization, so I wouldn't be surprised if GPU-Z says one thing, when the reality in the card is something else. It's certainly useful for general troubleshooting, though.

    You always need some VRAM headroom, and assuming a 3.5G actual limit, at 3.3G you're pretty close. 

    I can understand how different poses might change the memory requirements, depending on textures, vertices exposed, and other factors. 

     

    Post edited by Tobor on
  • ToborTobor Posts: 2,300
    marble said:

    Now I've tried the same scene with half of the content taken out and it is still refusing to render. Another scene with more content is working ok. Confused, I am.

    I've had this in a few scenes, and it's not really a RAM limit issue. Sometimes the scene database gets corrupted on its way from D|S to the card. CPU renders are okay, but GPU renders are more sensitive, it seems. In one case I had last year, there was some bad skin texture or shader setting. I never found out what caused it, but when I rebuild the scene from scratch, using the same textures and reapplying all the shaders, all renders then worked fine.

     

  • ToborTobor Posts: 2,300

    Just as an FYI, this line:

    out of memory (while launching CUDA ior stack renderer in core_cuda_host.cpp

    suggests the out of memory error is coming from pre-processing done for calculating the refractions in the scene. In programming, stacks are used as containers for similar objects, and it's possible for an error (in the source data, in the programming, both, other), for the stack to overrun its memory limits.

  • marblemarble Posts: 7,500

    On the 3.5 thing - I have read elsewhere on this forum that DAZ state that IRay can use the whole 4GB. 

    What you say about pre-processing makes sense because I have had occasions when I've set GPU-Z to show the High Water Mark which, sometimes, has not gone beyond 3.2GB. Now I do see a blip just as the render begins and then fails over to CPU but that does not get logged at exceeding 3.5 or even close. Oddly, I've had 3.3 render fine and then 3.1 fail. Go figure.

Sign In or Register to comment.