4.15 + RTX 3090 Anyone? Iray always falls back to CPU or crashes!
So, I'm struggling since 3 days now and don't know what else I could do. I tried out every nvidia driver and even reinstalled Windows 10. But doesn't matter what I do or which nvidia driver I use, Iray preview always either crahes or falls back to CPU or just stops working (if you for example make some changes in the scene, hide show thinkgs or surfaces etc). I got two "MSI RTX 3090 Gaming X Trio" GPUs (no nvlink bridge at the moment but very soon) and tested them with 3D Mark. Everything was fine with the GPUs. So, my only explanation is that Daz Studio 4.15 and RTX 3090 are still not compatible together!
I'm getting this error for example:
2021-01-18 03:26:18.194 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.26 IRAY rend error: Error synching on OptixPipeline event (CUDA error string: the launch timed out and was terminated, CUDA error code: 702)
2021-01-18 03:26:18.199 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.26 IRAY rend error: Error destroying OptixPipeline event (CUDA error string: the launch timed out and was terminated, CUDA error code: 702)
2021-01-18 03:26:18.199 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.26 IRAY rend error: optixPipelineDestroy(pop_ptr(m_pipeline)) failed: CUDA error
I also found this information:
cudaErrorLaunchTimeout = 702
This indicates that the device kernel took too long to execute. This can only occur if timeouts are enabled - see the device property kernelExecTimeoutEnabled for more information. This leaves the process in an inconsistent state and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched.
But how it must be done, I don't know. Does it mean one of my GPUs is defect (Cuda cores)?
Or this error:
2021-01-18 03:39:58.544 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Using iray core convergence estimate.
2021-01-18 03:39:58.544 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: No device specified or usable
2021-01-18 03:39:58.549 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering...
2021-01-18 03:39:58.549 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(359): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: No worker to render with: aborting render
2021-01-18 03:39:58.549 Iray Render error: Internal rendering error.
Does anyone have a solution? I'm working with DS for living so it's really a disaster for me since I already sold my Titans RTX and my work stagnates at the moment!
Comments
This may seem unrelated, but may not be and be worth checking out.
3090s are power hungry. Iray I found to have a unique power ramp up profile.
Make sure you have a PSU on a single rail (BeQuiet's from Germany are often multi-rail). And get one with lots of head room. Sudden power needs can push what should be good enough, to maybe not quite.
If your one GPU is power starved then that could cause enough delay.
Would start there as you did not describe your PSU. And from my own recent experience with DS-Iray, i can say just because your GPUs works flawless with everything else, does not mean you wil have same result with Iray.
Can you render with only one card selected? If both cards fail with the same error when selected alone I think you can rule out that one of your cards is defective and that it is a psu issue.
It would be highely unlikely that both cards are defective.
Moreover your error message suggests a driver issue.
Have you tried older/newer versions of the geforce driver (depending on what you have installed now)?
Is it the same with daz-studio.4.14 (I think that version already supported the rtx 3090).
At last resort I would install one of your gpus in another pc with daz-studio and look if I get the same problems (maybe you have your old rig lstill aying around).
I got a very good 1300W Seasonic PSU. My CPU is AMD Threadripper 1920X; RAM: 128 GB DDR4 Corsair;
I tried it also with a single GPU, also even removed one physically. And it was the same. I got a nvlink today and will try it with it.
From what I heard, the 1300W should be enough for two 3090. I mean they say 850W is enough for 1 3090. I've seen even successful tests with 600W PSUs! My 3090 has a max power of 370W. So, 850+370=1220W. It's tight, but should be enough. The problem is also, that because of that annoying corona time you can't get any PUS higher than 1200W in Europe. Only from US, but they got different power cords. Only this one would be available: "1500 Watt be quiet! Dark Power Pro 12 ATX 2.51 80 PLUS Titanium". But they say it hasn't enough cables (the 3090 needs a 12 pin plug! So, I need 24 pins!) and shouldn't be that great.
And I also don't think that the error "CUDA error code: 702" has anything to do with the PSU.
I tried every driver from 456.38 on and none of them worked. And I also wanted to return to DS 4.14 but you cannot do it anymore in the DIM and I also don't have the file. It's an absolute mystery for me why they removed the option to install older DS versions, especially if the brand new ones are buggy!
Even with only one GPU it falls back to CPU with the error code: "CUDA device 1 (GeForce RTX 3090) ran out of memory and is temporarily unavailable for rendering." Which is a joke since it's just one character an no backgorund in the scene. So, it can't be a PSU issue!
Is there really nobody out there who could help me? Is there a way except 3D Mark how to 100% check if the GPUs are alright? With 3D Mark everything was fine. And my mainboard can't be the issue as well, because my two Titans RTX were working absolutely fine before. But also with a very old driver (1,5 years old), because all newer drivers failed with IRAY.
So, more and more I'm sure that the problem is in Daz Studio and Iray itself! That 4.15 works unstable with RTX 3090! Which would be really a mystery how after 4-5 months these GPUs are already out, Daz Studio and Iray still didn't manage to make it run properly!
Normally a good 750w psu should be more than enough for one card. I think the problem were performance peaks. Igorslab (a profilic german hardware tester) reported that his psu (a top modell in the same wattage range like yours) gave out when reaching performance peaks and the computer turned off. And that was with only one card. But that was at release of the cards and got fixed with updated drivers. Also if your cards need to much power for the psu it should turn off your computer for safety reason. So I don't think it's a psu issue here. Also 1300w is more than enough for two cards.
OK, but what can be the reason for these issues except that Daz Studio 4.15 is not compatible with 3090 yet even with the latest driver?
What is in your scene?
I'm loading one of my character, with clothes and everything. Then in the Iray preview I'm playing around with the surface, changing clothes and then their surface settings etc. Sooner or later it falls back to CPU or if this option isn't available it either crashes or Iray just stops working.
But it actually doesn't matter which scene I'm loading, it happens anyway.
It would matter if it was happening because the scene really was running out of memory but, while not impossible, I think that's fairly unlikely now. Updating the Iray Drawstyle render data (by makng a change) and then updating again before it's fully caught up can, or could in the past, cause the render to drop the GPU and go to CPU(which then persists for the rest of the session).
Just to double-check? You have latest 3090 bios firmware updates via what must be their update utility? Cos checking MSI website shows Bios from Oct, which seems bit old for new tech.
Looking at threads in DazStudio Discussion subforum (as opposed this Tech Help subforum) there are several users that are happily rendering away with 3090 and based on their descriptions, pushing it to its limits without dropping to CPU. One mentions using latest DS 4.15.
You may be using it differently than others with all your mat changes. I did have this issue over a year ago, and at that time it was a DS issue, that has not reappeared. Am using my lowly 2080Ti with 4.15 doing all kinds of mat changes without any issues in lengthy sessions. So 4.15 seems fine for older cards.
Just a suggestion, but you may want to open a new thread over there with title being something like "Help: 3090 survey - Drop to CPU? yes or no?" Perhaps mod can assist with thread merge if duplicate threads are a big issue?
Impression formed over time is more 3090 users are there and less in tech section.
You might then be able to survey and see if others are having similar. But my impression reading comments of others so far, is that 3090s and DS is working fine.
Possibly NVlink will help?
Sorry I am still not updated to 3090s ( darn bitcoin craze & covid homebodies and PC parts shortage), so I can't compare and offer any help.
The Bios updates from MSI are just for "low noise" and "low temperature". I don't think that this will help in any way here.
Yeah, I also don't know why I got this problem and other say they don't After all I even reinstalled windows 10!
And which forum here do you mean exactly?
But for example it worked some time and then suddenly it crashes again with that very weird error in the log file.
Hmm so no new Bios updates via update utility. Yeah saw the ones on website were just for those 2 purposes. Was thinking maybe new updates available ONLY through your desktop update utility? Sounds like not.
https://www.daz3d.com/forums/categories/daz-studio-discussion --- further down below pinned threads.
Well good you got Win10 reinstall out of the way to eliminate that as a possible cause.
Looked through your log and compared with mine for startup. Nothing really stood out. Did not see weird error in your log. Did see alot of "fails" for resources to be loaded but unless that is constantly streaming can't see how that would affect memory in a big way. Which is what drop to CPU or crash is most often caused by in the 2 years been doing this. Maybe pull out 2 (0r 4) Ram modules and see if systems crashes differently? Computers can be so wierd how they conflict sometimes.
Are you monitoring system resource usage GPU VRAM & utilzation and also system RAM? What does it look like when you crash?
Looked online for a utility to release VRAM. All i found was linux based commands. So that Cuda failure would need either DS shutdown/restart or reboot for a reset.
Everytime you manipulate the scene in iRay Preview you're liable, rather DAZ Studio, to be leaving allocated memory unfreed until you eventually run out of free memory, 24GB video RAM or not
But this never happened before with any of my previous GPUs! I had two Titan RTX before and they worked flawless doesn't matter how I stressed out the Iray preview! Even though they worked only with a very old nvidia driver.
When it crashes, then just DS closes. I also got an effect (see the image below) that after some time the Iray preview starts looking like damaged. Everything is distorted as if your GPU is damaged. But it's only the Iray preview and not my whole screen. Then I hide the shirt and it works normal again but only with one GPU working. This is just an example what sometimes is also happening. So, it's totally weird, as if DS does things when and how it feels like.
Like I said. I always had the most powerful GPU series of the year. 1080i, 2080ti, Titan RTX and I never experienced such issues!
And I also don't think that this is a PSU problem like some mentioned. Because I tried it also with only one card (removed the other physically) and it was the same.
Ever since 4.12 Daz3D has been effectively acting like it has a HUGE VRAM leak. if you do test renders and look at the VRAM you'll note it doesn't drop to the original start point. this does make the render fall back on CPU EVEN if all ways to stop CPU fall back have been used.
3080's also crash and refuse to render with poly heavy scenes this seems to be also ever since they changed the Core system to cater to the RTX cards. guess someone forgot to put a Space or a "," somewhere.
You really have to question the latest versions as being Great leaps forward when old GTX cards using older versions could render scenes with 19 + characters and environment while the latest Version with the Latest twice as powerful cards Crash at 12 characters with less demanding environments. But WOW you can do a portrait of a single character in no time Guess the only people being catered to are those making the Advertisement Pics for Daz store products!
Edit: Here is an example it took an RTX a double pass to create this while a GTX and 4.09 could do it in one go all characters in the same scene instead of half&half
Since a few days I have issues with Iray and rendering with my two MSI RTX 3090 Gaming X Trio! I tried out every nvidia driver and also reinstalled windows. But with every driver I got some other issue. Iray preview either stops, freezes or jumps to CPU. Or with 461.09, Iray preview's running for a couple of seconds and then the preview display looks weird (see image 1). Then I hide a prop and it keeps on working but with just one GPU. During rendering, after around 2000 iterations the render is suddenly completly white or a huge error look (see pic 2). But this happens only with driver 461.09!
With the other drivers it either freezes, crashes or jumps back to CPU. So, the log file errors are different with every driver. There're some examples below. I also got Cuda core error 702 and 999 with some drivers. 702 with the later ones and 999 with the older ones. And yes, this happens with every scene I load, doesn't matter how big it is!
I got a 1300W Seasonic PSU. I tried it also with 1 GPU and also put 3 different PCIe cables into the GPU slots. But it was the same. So, I can't imagine that they don't get enough power. I really don't know how some people can say 3090 works great! For me, Daz Studio still doesn't work stable with the 3090s, with none nvidia driver! I did a 3D Mark benchmark test and everything was fine. So, the GPUs seem to be OK.
But maybe those who're working with a 3090, or even better with 2, could tell me what their settings look like, which driver they use and what could be the reason that I have such huge issues to make my 3090s run with DS!
Although I haven't had the exact same issues, my MSI 3090 Ventus crashes frequently in DS 4.15 regardless of driver. It's not consistent, but I've also found it randomly falls back to CPU or hangs on the 'Compiling Shaders' at the beginning of a render. I didn't have this issue prior to upgrading to 4.15 - it was amazing how stable it was on 4.14.
I use a quad GPU setup with 2 3090 FEs and 2 RTX Titan FEs with the minimum drivers DAZ needs (451.48 SD) on DAZ 4.15 Beta. Has worked without issues. Iray preview is smooth and bottlenecked more by load times. Stills and even long movie renders work fine. Anything specific you'd like me to check? I haven't messed with Iray occlusion planes.
I did initially run into issues when I first installed the 3090s (migrated from just the Titans) so I had to experiment with drivers. I do recall not being able to use anything above the DAZ minimum driver version, though DAZ has been updated many times since then & I haven't tried again. I've also not tried the non-beta version of DAZ since the beta was the only thing that worked on my Titan for a while (the non-beta eventually caught up).
3d Mark worked fine out of the box in any supported driver. So it's totally possible to clear gaming benchmarks but not have DAZ or other 3D software work. I also did have to deal with a bad PCIe riser cable but GPU stress tests will usually sniff those out. Iray render power consumption can be much higher than gaming, especially with multi-gpu, but you can control for that by testing only a single GPU at a time in DAZ.
Edited to correct minimum driver version
Thanks - I think I might try the Beta to see if the issues remain.
I had two TItan RTX before and they worked great but also only with a very very old driver. And like I said, I tried it also with 1 GPU (physically removed the other) and it was the same!
I think I already tried the SD driver 456.71 but I will give it another go and look if it helps.
Just tried to render with driver 456.71 (4.15 Beta) and it jumped only after around 400 iterations to CPU again with that CUDA error code 999 (see log file)!
It really could be that the MSI 3090 Gaming X Trios are too powerful for Daz Studio, since you're using the much weaker founders edition.
The more powerful 3090s (I got 2 MSI Gaming X Trio) are still not compatible with DS and are working extremely unstable with every nvidia driver! Crashing, freezing, falling back to CPU etc. in Iray preview and during rendering! When are you gonna finally make it work properly???!!!
If you set the viewport to "Texture Shaded", does it perform the actual rendering with IRAY?
The thought behind the question is, for most users using IRAY viewport is too slow to work with, therefore it is possible that some bugs have gone unnoticed, especially if nobody has had the exact combination of hardware, software and drivers (all of them) that you have.
Tracking down the culprit in PC environment is not easy due to the amount of variables.
It's always on "texture shaded" when I start the render. It has never been different. Currently I got only one explanation: DS and Nvidia still didn't manage to make very poweful 3090s work stable with the program. I really don't have any other explanation!
I don't see how a GPU being "too powerful" could be bad for Studio except stressing your hardware for power consumption. I'm also pretty sure my quad GPU config is at least as fast as 2 after-market 3090s.
If you've tried a single card then maybe its not PSU stress. Have you checked for power issues on the cards? Egs., is it factory overclocked & can you downclock? When I tried OCing my FEs they'd run fine in games or 3D Mark but not for an Iray render. You can monitor temps and power consumption to make sure that's not an issue.
Other than that, guess I don't know. I am usually able to solve my issues with driver version selection.
Just checked: I'm at 1695MHz on the FE vs 1785MHz on the Gaming X Trio for core clock. You could try backing off and see if it helps. I know for a fact I can't run my FE at 1785M in DAZ.
I got these setting in Afterburner. I already tried to minimize everything on the minimum. It also didn't help.
How exactly I can set it back to 1695 MHZ?
In the afterburner you cannot set a precise clock frequency. I tried to move it down to the minimum but it didn't help. Is there another way how to set the clock frequency to exact 1695 MHZ? Biut I checked the afterburner and during the render it was under 1600 MHZ and still jumped back to CPU with CUDA error code 999.
Maybe it would also help to know what this 999 error code means and maybe here to start digging?
Not the same problem, but shows how not all test programs strain the system enough to find the problems.
The link leads to the post with the "Heureka" moment.
https://www.daz3d.com/forums/discussion/comment/6215226/#Comment_6215226
Afterburner displays info differently depending on the skin. You should be able to see the clock frequency in egs., the Mystic skin (go to user interface in settings to change it). You don't need to set the exact frequency. Just experiment with negative values on the "core clock" slider.
The units are usually in MHz for core clock. So factory frequency + delta = final clock. You can look up the factory frequency from the website you got your card and the delta is what you set in Afterburner (can be negative).
I'm just guessing about the root cause here. I've no idea if MSI tests their cards with due diligence and content creation workloads before delcaring the OC stable. But if you've tested both your cards individually, it's probably not a one-off error.