prevent render from switching to cpu?
just lost a whole bunch of work... I was creating a working headlight for a truck model...
running 500x400 renders taking about 40 seconds apiece will testing light combanations.
---
the all of a sudden it's rendering on the cpus... and worse .. appears to have hung up... because dual xeons will render pretty quick by themselves...
----
is there any way to make sure Daz doesn't do this ..(cpu is not checked in the advanced settings tab.
----
does this kind of a problem happen very often?
----
daz finished a render ... then decided it had some kind of error and started to render again
and decided to use the cpu which since I wasn't planning on using the cpus .. I didn't have the limiter turned on
but it also hung up halfway throught the render and using 100% of the cpus made stopping it a read pain...
-----
what happened was an error condition
and then a code decesion to switch to CPU...
this requires an interrupt message and asking the user if they want to continue or stop the render.
log file:
2017-06-30 18:45:21.889 Finished Rendering
2017-06-30 18:45:22.035 Total Rendering Time: 53.46 seconds
>>>>>>>>>>>>>> finished a render and then
2017-06-30 18:45:40.594 WARNING: QEventDispatcherWin32::unregisterTimer: invalid argument
2017-06-30 18:45:46.127 Rendering image
2017-06-30 18:45:47.528 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Updating geometry.
2017-06-30 18:45:47.529 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Using built-in ray tracing (1.0.0).
2017-06-30 18:45:47.529 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Importing geometry.
2017-06-30 18:45:47.861 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Geometry import (1 object with 753k triangles, 1 instance yielding 753k triangles) took 0.331399
2017-06-30 18:45:47.861 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Updating materials.
2017-06-30 18:45:47.883 Iray INFO - module:category(MATCNV:RENDER): 1.0 MATCNV rend info : found 109 textures, 0 lambdas (0 unique)
2017-06-30 18:45:47.889 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Emitter geometry import (5 light sources with 5682 triangles, 1 instance) took 0.00s
2017-06-30 18:45:47.889 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Updating environment.
2017-06-30 18:45:48.256 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Updating lens.
2017-06-30 18:45:48.256 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Updating lights.
2017-06-30 18:45:48.257 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Updating object flags.
2017-06-30 18:45:48.257 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Updating caustic portals.
2017-06-30 18:45:48.257 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Updating decals.
2017-06-30 18:45:48.259 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Allocating 1 layer frame buffer
2017-06-30 18:45:48.267 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Using batch scheduling, architectural sampler disabled, caustic sampler disabled
2017-06-30 18:45:48.267 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY rend info : Initializing local rendering.
2017-06-30 18:45:48.267 Iray INFO - module:category(IRAY:RENDER): 1.0 IRAY
>>>>>> bingo >>>> rend info : CPU: using 32 cores for rendering <<<<<
why in the hell does this happen. Cpu is not checked as a rendering option.
Comments
If the Graphics Card RAM is insufficient it will drop to CPU regardless of being checked or not.
Sometimes I've noticed it happening after a few renders, and I close what I have open; it does seem to happen more than it used to - either a new feature or a bug. :)
If the GPU can no longer render it will drop out, and if the CPU is not ticked, rendering should simply stop. If you're seeing otherwise, first determine that the log message isn't simply bogus, and that the CPU is in fact not rendering. If it does appear that the CPU is rendering despite not having it selected as an option you should open a support ticket with Daz.
To avoid losing work, be sure to save more often, especially right before you start a new render. It's a hassle to always wait for the resave, but it's less time that rebuilding what you already did.
As Nicstt noted, you're probably seeing the effects of a clobbered scene database by successive tweaking and re-rendering. You'll need to quit and Restart D|S, and possibly even reboot to reset the GPU. The issue may be a misbehaving shader or geometry, so it may not happen on every scene.
When you post, be sure to indicate the D|S version you are using.
this batch is Daz not clearing the gpu... closing daz immedicatly resets it...
---
the 1050 only has 4g so about 5 little renders top it off...
---
but the cpu is turned off and it does it anyway...
===
I now make sure the limiting program is holding daz at -20% on cpu usage to avoid it taking over the machine.
---
previous card was a 980 ti with 6g and I"m sure I did many small tests without the issue...
I did have to upgrade from 492 to 494 because the render engine wouldn't recognize the 1050 to work with at all (
---
an ususal feature is it's the little renders that are stacking ones only using about 500 or 600 megs of gram
but the big ones that are maxing the card out (render i"m doing now is at 4060 of 4096 and I actually restarted it twice and the base amount of about 600m never bumped up inbetween.
---
the one that cost me the work was a really off the wall one as noted above ... it finished the render ... then had an error condition and took off with doing it again.
---
and it was hung in some kind of error because generally the gpu is at best twice as fast as the cpus so a little 500x400 test render should only be about 5 minutes even on the cpus...
----
switching during a render rather than ending because of a shortage of g-ram would make sense..but just deciding to start the next render in cpu without saying "hey, I can't use the card, continue or save and reload"...
---
if the render engine can be verbose... why not the main program?
---
a "save on render" preference setting would be nice...
Simple answer... Nvidia wrote Iray - more coders than DAZ. DAZ is a small company.
the log file is produced by daz... I'm sure the daz coders produced the data found in the log file... which is what is rendering box presents while rendering.
===
if you generate that data and "print" it in the log file... you can certainly add an if statement with conditions like if print line = switching to cpu and cpu not enabled then print screen "daz want to use the cpus" ...
--
that kind of coding is not rocket science ... writing the program itself is...
1. Iray is produced and coded by Nvidia. Not Daz3d
2. Iray is a rendering engine plugin, just like 3Delight. It is not coded by Daz3d, all they did was write the interface for Studio to talk to the render engine.
3. When there are no GPUs remaining in your preferences that can render the scene, Iray drops down to CPU as a failsafe without asking for any user confirmation. (probably as a reliability thing for render farm computers)
4. Daz3D is probably not alowed to change the Iray code, as they do not own the source code for Iray.
we're not talking about changing the code...
I doubt if the messages that come into the log file are propriety for nvida.. particulary as they include all the info daz generates as it opens a file...
---
so reading the report as it comes into the log file and then generating approriate error messages is just something that needs to be done...
clear out your log...
then start a new scene... load some figures...
then do a render ...
---
Iray doesn't give a damn when you opened the file ... what i was called ... what morphs were loaded...
the log file is daz file...
---
it includes what the iray engine sends it, but lots of other stuff
---
and there is lots in there that could generate error messages.
like not being able to load certain objects or textures ... but wait, it does tell us when that happens so we don't have to read the log file...
---
but when I was getting black renders because the iray engine could not use the 1050 card... it even send a message to daz.(in the log file) . "I can't use that card". but daz didn't bother sending that message to me.
---
you can render with 3delight and have the same kind of messages in the log file.
--
as for the render engine... daz starts it, can pause it, and can end it with a cancel command.
---
nvida may have written the code that does the rendering but daz gives it data and tells it what to do.
and receives the data...
The log entries during a render are generated by callbacks from within the Iray libraries. They seem to be verbatim what Iray is returning, and not modified to make them more user friendly. That would probably be a huge help for a lot of users who find the messages cryptic and incomplete.
I'm still not clear whether the CPU is actually doing any rendering work, or if it's a bogus message from Iray. You'd normally get an uptick in activity at the beginning and end of a render, as both involve processing that does not involve the GPU. Example: building the scene database is done entirely in the CPU, and at that, usually with only a single core per physical CPU. Canceling a render is, or at least has been, a source of CPU swamping. It can take several seconds or even minutes for the machine to come back if the CPU was used for rendering. It's been this way since 4.8.
I also have a dual Xeon machine and have it set up so that only my GPU renders. I have never seen this, nor do I get hangups post rendering. I'm still using the last 4.9 beta release, though, and have not upgated to the latest public release.
Strange...it seems like I might be getting the same problem.
I have a GTX 1070, and noticed it was taking very long on an interior scene. And just now I noticed using CPUID that the CPU's were pegged but the GPU was almost 0% utilization during a render. And I've always un-checked the CPU render option, only my GPU.
And at one point in this same scene during a render it started then just quit. Did that a couple of times.
Strange...
Last week I had this problem. Frames 1 and 2 of an animation took two minutes each; frame 3 rendered for (I think) 90 minutes before I saw what was going on and stopped it.
After that Iray renders wouldn't use the GPU at all until I quit and restarted DAZ Studio.
Two possible problems occur to me:
I might be able to submit a log file, if we think the messages could be helpful.
Okay, in my case I think it may be that my GPU is running out of memory. Go figure. GTX 1070 with 8GB and that's not enough.
During the render the Iray verbose said "geometry memory consumption" was about 350MB, so I figured I'm good. But now when I check CPUID it says that, during the render, Memory Utilization of the GTX 1070 is hitting 95%.
So I killed D|S, rebooted, and now it's getting up to 90%, but the GPU is doing it's thing at 98% utillization, and the CPU's are at around 30%. So it sounds like I'm operating at the ragged edge, and with more tasks going to the GPU (Blender, whatever else is open), maybe that's the breaking point.
So this scene, which was taking 40 minutes to get past the 0% render percentage, is now there at 4 minutes. Not 40.
So I guess the lesson learned: you can never have enough memory, even in your GPU. Though if anyone knows if this 8GB usage is reasonable or if there's something else going on I'd sure appreciate it.
Attached is the CPUID info during the render.
Also, to Inkubo's point, maybe the GPU memory isn't being cleared when it should, so with a fresh start you're okay, but after a while stuff builds up in memory and you hit the breaking point because the previous stuff wasn't flushed. That sure could have been the case with my scene. Do a partial render, kill it, start again, kill it, and so on.
It drops to CPU if not enough available RAM on the card; personally I agree, a warning that its dropped to CPU when CPU is not checked would be appropriate.
... writing a line in a debug text file isn't user friendly.
Depends, but a small scene with two or three characters can take up quite a bit. It depends on textures far more than geometry; so too does texture size.
What surprised me was I can load a Stonemason city with say 5 characters added and no problem. But here I just have a single house model with furniture and it breaks the bank.
Strange.....
Yup, that was it. I cancelled a render while GPU Memory Utilization was at 83%, and it stayed there even after the render had stopped. I waited a bit, then re-ran the render, and as soon as it gave me the first grainy iray preview BAM it went to all CPU's and pegged them.
Another lesson learned....
Monitor your GPU memory utilization.
I guess they hold everything in memory to make subsequent renders faster, which is good. But at the cost of memory.
8GB should be enough for a fairly complex scene. You could look at one of the scripts to resize images, and in the case of https://www.daz3d.com/scene-optimizer make other tweaks, to try to deal with the size. Also check the Render Subdivision setting (in Parameters) and Displacement Subdivision (in the Surface settings for surfaces using the Iray Uber shader) as each object will be dividided, throughout, to the maximum setting on all its surfaces and in Parameters.
Yes, if the GPU runds out of RAM you will probably need to restart DS to free it up again.
Excellent. Thanks Richard.
It's funny I went out to do some chores, my computer just sitting there doing nothing, no renders, D|S just sitting there, and when I get back my GPU Memory Utilization still at 84%...
So yeah, watch your memory.
EVGA's Precision X app lets you set various monitors in the taskbar and works with non-EVGA-branded Nvidia cards. You can have it display GPU usage (displayed as % ), temperature (in Celsius), and VRAM usage (displayed in Megs), as well as Bus and FrameBuffer usage, but I've not seen those change during renders.
As well, expanding the little Render popup will give you some details on GPU usage just before it startes counting out the iterations, such as Lights, Mesh, and Texture numbers. If anything exceeds the VRAM of any of your GPUs, it will dump right to the CPU.
Switching between saved scenes will also stack in the VRAM, I've found. It's best to close and restart D|S between scenes to flush that out. The Script that flushes memory only appears to work on system RAM. Would be nice if it worked on VRAM as well.
Smaller scenes (the single room mentioned above) may have higher-res textures than the "big city" scene - the Reading Room uses 4K textures, IIRC, and that with 3 figures, Iray Worlds skydome, and Iray emissives for all the light bulbs in the scene maxes out all 3 of my GPUs (Pascal Titan, Titan Z, and a 4GB 980) and takes over 2 hours.
However, none of that explains how or why it ended/stopped/completed the GPU render cycle and then went into a CPU-based render. Are you running a post-render script that calls for a second render?
Somethings still strange here...
I went thru and deleted everything except for one room. No characters, just some walls, ceiling, floor, and some furniture.
And when I load from scratch and do a render the GPU memory utilization on an 8GB GTX-1070 goes up to 60%. That just doesn't make sense. Only a bed are high resolution, with Render SubD set at 1. All the rest are base resolution. And I'm certain none of the surfaces have displacement.
How can such a bare scene use 60% of an 8GB GPU?
I use CPUID HWMonitor, a free app that monitors all of that GPU stuff (volts, temps, fans, power, clocks, utilization), hard drives (temps, utilization), CPU (temps, clocks, utilization) all in one display. So you can watch stuff bounce between your CPU and GPU, and watch the drives spin up and so on.
Okay, attached is a render of what's left in my scene. It caused the GPU memory utililzation to go to 51% throughout the render, and the GPU was fully utilized (ie, no CPU rendering).
Now I have no clue what could be in this scene that could cause the 8GB GPU to use half its memory to store textures or whatever else. At most the floor might have a 4k texture (what is that, 16 MB total or something?).
I don't get it.
Start D|S. GPU Memory Utilization = 9%.
Create primitive sphere. Create primitive plane. Render.
GPU Memory Utilization on a GTX-1070 w/ 8GB of memory is 33% throughout the render.
See attached render.
I really don't get it. It appears that Iray in D|S has a basic overhead requirement for even the simplest scenes of something like 2-3 GB?
And I used another GPU monitor (GPU-Z), and it showed exactly the same thing. GPU Memory used was 2.6 GB just to render a primitive sphere and plane. The attached shows the charts of monitored stuff as I load the scene and render it.
Iray does the computations also using your GPU memory. The rendered image is also taking memory. I don't know if it holds a preset amount, but it may. If your card is just rendering it's using all it has. If it's also your display's card, the OS is involved and holds memory for system functions.
I took a look at the Iray verbose stats, and it shows the following for my big scene that was on the ragged edge of memory consumption:
Texture Memory Consumption: 10.7 GB
Lights Memory Consumption: 4MB
Materials Memory Consumption: 1.45 MB
Geometry Memory Consumption: 352 MB
So apparently there's more texture memory consumption than the GPU memory can hold. I guess it swaps to my 48GB system memory?
For anyone posting their stats it is VERY helpful to also note the version of D|S you're using. There was a recent new public build, a long string of 4.9 builds and betas, and of course there's 4.8, and many of them have used different Iray versions. It's not practical to even cross-reference problems if the program version is not stated.
That texture consumption value is prior to compression. At 10G, though, it does point to the possible need to manually tweak your textures, if possible. For example, you can make the textures of far-off characters much smaller, without losing resolution. Not every character in a scene needs 4K textures. Remember that Iray does not instance textures, even if the files are the same. It only instances geometry.
Oh, okay. I'm using 4.9.4.117 Pro, 64 bit. I think it's the latest public build, not a beta.
Does that version number make a difference to memory requirements for Iray?
Thanks.
Honestly I don't think any of my textures are anywhere near 4k. Maybe some store-bought stuff is, but I kinda doubt it. But even without characters there's a ton of memory being used, and I'm not sure why.
In your original post in the other thread, you said it was 4 rooms. What's in those rooms? Can you delete them? Iray loads all into memory, hidden or not. That's why Stonemason gives you the option of not loading certain city blocks, etc. in some of his sets.