render progress suddenly reversing

Hi,

twice now in the last week (and never in the years before that) I have seen a render going from e.g. 70% completeness all of a sudden back to to e.g. 30% completeness. From looking nearly finished, it suddenly became much more grainy again. That's of course a big time waster. Any idea what this is due to?

thanks

Comments

  • dijituldijitul Posts: 146
    edited December 2019

    I've not seen this happen before (where it goes back to being grainy).  However, the render will change percentage if you adjust certain settings during the render such as quality.  Could you provide more details such as which render engine you're using (e.g. iray or 3delight), if it's a specific scene/project or any project you load, and perhaps a fresh log file of a misbehaving render?

    Post edited by dijitul on
  • jstiltonjstilton Posts: 134

    thank you. i didn't adjust anything during rendering. it just suddenly went in the opposite direction.

    i'll post a log file if it happens again and i can catch in the log (right now i wouldn't know where to find it in the log as it happened quite a few renders ago)

     

  • I used version 4.12 and the whole program crashed at the end of the render. This has never happened with previous versions!

    Don't know how to solve!

  • jstiltonjstilton Posts: 134

    So this thing (rendering process suddenly reversing) still happens regularly. The last time, today, I think that I know more or less where in the log it occurred. I think you can see it in the fragment below. Anyone know what's going on?
    Thank you!
     

    2020-08-17 21:58:01.119 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:58:08.443 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00691 iterations after 4940.534s.

    2020-08-17 21:58:08.505 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:58:15.982 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00692 iterations after 4948.075s.

    2020-08-17 21:58:16.040 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:58:23.356 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00693 iterations after 4955.448s.

    2020-08-17 21:58:23.404 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:58:28.002 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Device statistics:

    2020-08-17 21:58:28.002 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CUDA device 0 (GeForce GTX 980 Ti): 3357 iterations, 29.071s init, 3983.652s render

    2020-08-17 21:58:28.041 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CPU: 555 iterations, 29.308s init, 3983.423s render

    2020-08-17 21:58:30.512 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00694 iterations after 4962.604s.

    2020-08-17 21:58:30.554 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:58:30.554 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Available GPU memory has increased since out-of-memory failure. Re-enable CUDA device 0.

    2020-08-17 21:58:37.580 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering...

    2020-08-17 21:58:37.586 Iray [VERBOSE] - IRAY:RENDER ::   1.6   IRAY   rend progr: CUDA device 0 (GeForce GTX 980 Ti): Processing scene...

    2020-08-17 21:58:37.616 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.31  IRAY   rend error: Unable to allocate 63091776 bytes from 51225395 bytes of available device memory

    2020-08-17 21:58:37.617 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.6   IRAY   rend error: CUDA device 0 (GeForce GTX 980 Ti): Scene setup failed

    2020-08-17 21:58:37.631 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.6   IRAY   rend error: CUDA device 0 (GeForce GTX 980 Ti): Device failed while rendering

    2020-08-17 21:58:37.680 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00695 iterations after 4969.767s.

    2020-08-17 21:58:37.734 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER ::   1.0   POST   rend warn : renderer iray has no more devices available. Postprocessing falling back to CPU.

    2020-08-17 21:58:37.752 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Updating backplate.

    2020-08-17 21:58:37.767 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:58:37.773 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering with 1 device(s):

    2020-08-17 21:58:37.773 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CPU

    2020-08-17 21:58:37.773 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering...

    2020-08-17 21:58:37.777 Iray [INFO] - IRAY:RENDER ::   1.6   IRAY   rend info : CPU: Scene processed in 0.001s

    2020-08-17 21:58:37.876 Iray [VERBOSE] - IRAY:RENDER ::   1.6   IRAY   rend progr: CPU: Processing scene...

    2020-08-17 21:58:45.349 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00001 iterations after 7.572s.

    2020-08-17 21:58:45.423 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:58:52.176 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00002 iterations after 14.399s.

    2020-08-17 21:58:52.240 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:58:59.039 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00003 iterations after 21.262s.

    2020-08-17 21:58:59.100 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:59:06.524 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00004 iterations after 28.748s.

    2020-08-17 21:59:06.621 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:59:14.000 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00005 iterations after 36.223s.

    2020-08-17 21:59:14.036 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:59:21.353 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00006 iterations after 43.575s.

    2020-08-17 21:59:21.389 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.0   IRAY   rend warn : The 'iray_optix_prime' scene option is no longer supported.

    2020-08-17 21:59:28.641 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00007 iterations after 50.864s.

  • It looks like it switched to CPU, which is always slower.

  • marblemarble Posts: 7,500

    Never seen this message in the log ...

    Available GPU memory has increased since out-of-memory failure. Re-enable CUDA device 0.

    So maybe something you were doing on the computer other than rendering in DAZ studio was grabbing and releasing VRAM? I always thought that once it has dropped to CPU it would never return to GPU but this log seems to suggest switching back and forth. 

  • marble said:

    Never seen this message in the log ...

    Available GPU memory has increased since out-of-memory failure. Re-enable CUDA device 0.

    So maybe something you were doing on the computer other than rendering in DAZ studio was grabbing and releasing VRAM? I always thought that once it has dropped to CPU it would never return to GPU but this log seems to suggest switching back and forth. 

    Same here.. how can we fix this?

    Screenshot_846.jpg
    591 x 346 - 37K
  • nicsttnicstt Posts: 11,715
    marble said:

    Never seen this message in the log ...

    Available GPU memory has increased since out-of-memory failure. Re-enable CUDA device 0.

    So maybe something you were doing on the computer other than rendering in DAZ studio was grabbing and releasing VRAM? I always thought that once it has dropped to CPU it would never return to GPU but this log seems to suggest switching back and forth. 

    I've had it jump back and forth about five or six times; it is annoying, or at least it was.

  • nicsttnicstt Posts: 11,715
    marble said:

    Never seen this message in the log ...

    Available GPU memory has increased since out-of-memory failure. Re-enable CUDA device 0.

    So maybe something you were doing on the computer other than rendering in DAZ studio was grabbing and releasing VRAM? I always thought that once it has dropped to CPU it would never return to GPU but this log seems to suggest switching back and forth. 

    Same here.. how can we fix this?

    Move to Blender. surprise

    ... Joking aside; if the VRam is on the cusp of being sufficient, I expect it is tough to avoid - especially if the card is used for other things; and with consumer cards, even when you don't use it for monitors, Windows presumes it can be at any time.

  • PerttiAPerttiA Posts: 10,024

    "Unable to allocate 63091776 bytes from 51225395 bytes of available device memory"...

    If you count the numbers that's 60GB from 48GB... Which Nvidia GPU has over 48GB's of VRAM?

  • mclaughmclaugh Posts: 221
    edited August 2020
    PerttiA said:

    "Unable to allocate 63091776 bytes from 51225395 bytes of available device memory"...

    If you count the numbers that's 60GB from 48GB... Which Nvidia GPU has over 48GB's of VRAM?

    Uh ... last I checked 1 GB = 1,073,741,824 bytes. 63,091,776 bytes/1,073,741,824 bytes/GB = 0.0587588 GB = 60.169006 MB.

    Post edited by mclaugh on
  • PerttiAPerttiA Posts: 10,024
    mclaugh said:
    PerttiA said:

    "Unable to allocate 63091776 bytes from 51225395 bytes of available device memory"...

    If you count the numbers that's 60GB from 48GB... Which Nvidia GPU has over 48GB's of VRAM?

    Uh ... last I checked 1 GB = 1,073,741,824 bytes. 63,091,776 bytes/1,073,741,824 bytes/GB = 0.0587588 GB = 60.169006 MB.

    Sorry about that... It's been 30 years since I graduated and the effects of aging are starting to show... Although, if the available VRAM was only 48MB, what else was running at the same time?

  • mclaughmclaugh Posts: 221

    Just speculating, but:

    2020-08-17 21:58:37.680 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00695 iterations after 4969.767s.

    2020-08-17 21:58:37.734 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER ::   1.0   POST   rend warn : renderer iray has no more devices available. Postprocessing falling back to CPU.

    2020-08-17 21:58:37.752 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Updating backplate.

    2020-08-17 21:58:37.773 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering with 1 device(s):

    2020-08-17 21:58:37.773 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CPU

    2020-08-17 21:58:37.773 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering...

    2020-08-17 21:58:37.777 Iray [INFO] - IRAY:RENDER ::   1.6   IRAY   rend info : CPU: Scene processed in 0.001s

    2020-08-17 21:58:37.876 Iray [VERBOSE] - IRAY:RENDER ::   1.6   IRAY   rend progr: CPU: Processing scene...

    2020-08-17 21:58:45.349 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00001 iterations after 7.572s

    suggests that the fallback occurs during post-processing or when DS switches from rendering to post-processing. If the render data is cachd in the GPU VRAM rather than system RAM, the CPU may not be able to access it, and therefore needs to re-render the parts previously rendered on the GPU, in wich case:

    2020-08-17 21:58:28.002 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CUDA device 0 (GeForce GTX 980 Ti): 3357 iterations, 29.071s init, 3983.652s render

    2020-08-17 21:58:28.041 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CPU: 555 iterations, 29.308s init, 3983.423s render

    could explain why the render is "only" dropping back to ~30% rather than back to 0%.

  • AndySAndyS Posts: 1,438

    Hi,

    I experienced a similar behaviour. It occurred on animation rendering.

    Normally I can render over-800-frames animations with a constant VRAM occupation (up to over 80%). All on GPU - no problems.
    But there are certain constallations where iray renders some frames normally and suddenly re-calculates the whole scene. This of cause has the same effect as if you start an independant second render. Now the GPU is already occupied by the previous render and as a consequence iRay quits down to CPU for all the next frames of that same scene.

    This looks pretty much like a bug. It is not caused by any other action of the PC. It is 100% reproducable. I noticed DAZ suddenly adding some additional dsf files or morphes, if I remember right. But I'm not expert enough to create a corresponding bug-report.

Sign In or Register to comment.