Rendering with 12 GPUs (Not enough CPU Threads)

xdeome

Im not sure if this is a dumb question, but does DAZ/Iray need a powerful CPU to render with 12 GPUs? Currently I have 12 1080 ti's and my CPU is a i5-6400. When I go to try and render , I get an error in the logs saying not enough CPU threads so it would only render with 4 of them. So does it work like, 1 GPU per thread or something? I would also currently like to add that i had Optix Prime Acceleration Enabled. Not sure if that matters. If anyone is more knowledable on this matter I'd really appreciate your help.

WendyLuvsCatz

did you make sure the CPU was unchecked?

not sure how many people own that many cards that could answer, very impressive

bytescapes

xdeome said:

... Currently I have 12 1080 ti's ...

12??!?!?? Now we know who's been buying all the GPUs and driving the price up -- it's not the cryptominers at all, it's xdeome! 

xdeome

bytescapes said:

xdeome said:

... Currently I have 12 1080 ti's ...

12??!?!?? Now we know who's been buying all the GPUs and driving the price up -- it's not the cryptominers at all, it's xdeome! 

Well. I'll admit i was inspired by the idea of cryptominers to instead make use of it for rendering. But trust me I,ve bought them at inflated prices. But anywho, I do indeed have CPU unchecked.

SimonJM

Don't envy the leccy bill, but at least no central heating would be needed... 

Dim Reaper

This is probably not an area that many people can help with due to most of us not having that many GPUs, but if it is working with 4 GPUs then it does strongly suggest that each GPU needs its own CPU thread.

I did a google search for your processor and what seems strange( for an i5 processor) is that the Intel page lists it as # of cores: 4, # of threads: 4.  I had thought that they i5 processors all had hyperthreading, especially a 6th generation one like yours, but apparently not.

I do hope that you find a solution - I would love to see some render times on all 12 GPUs.

Bryan Steagall

hi xdeome

Doing a google search, I did find some places where they mentioned 1 thread per gpu, while in others they said it didn't make a difference, now, this was not for iray, but for other things like gpu mining.

I found this link from Nvidia, which seems to say (it is technical) that you can run multiple gpu's off one cpu thread http://www.nvidia.com/docs/IO/116711/sc11-multi-gpu.pdf

How are you managing all the cards? I'm assuming you are doing a riser? (as most motherboards will only support I believe 3-4 cards), there is a limitation on the # pcie lanes you can effectively use on some Intel chipsets (according to what I read,  LGA1155 or 1156 CPU only has 16 PCIe lanes . so consider that  too..) any other card you are using that is pcie will take one up (even embedded cards)

You might want to make sure all your bios/mobo/chipset drivers are all updated..  I also read something about an NVIDIA nf200 chip that will allow additional cards to be supported, as it takes over the pcie bus to some extent.

xdeome

Thanks everyone for replying! Like I said in my previous post, I was inspired by CryptoMiners so i bought a Motherboard that was interesting to me when I saw it, and that is that it had 19 PCI-E Slots. 1 of which is @16x and the rest at @1X. If i remember reading somewhere, that it didnt matter if it was 16x/8x/4x/1x for rendering. (Everything I am saying is based on some googling, correct me if I am wrong) So all of the cards I have take up 1x lane. I left the 16x slot empty. so, effectively, i am using only 12X lanes. And the i5-6400 supports 16 Lanes. 

I use to be able to render with 7 GPUs on my main computer which has a Threadripper 1950x and it worked fine using a couple of risers and even a PCI-E Splitter which takes a 16x Slot and turns it into 4 Sepearate 1x Slot. However, I wanted to create a seperate workstation, so i made a 12 GPU computer which only has a i5-6400 processor. 

Not sure if I need to have a better processor like a i7-7700k or 8700k or even the more expensive ones.

WendyLuvsCatz

we

l multithreading is only supported by i7 not i5 as I upgraded my mobo and doubled my threads to 8 with 4 cores

it was no faster though CPU wise just get more render tiles in programs that show them.

Utterly no idea about the GPU's though, maybe the Nvidia forums the better place to ask.

xdeome

For anyone whos interested, i think i can confirm that 1 GPU needs 1 Thread. I upgraded from a i5-6400 with 4 threads to an i7-7700k with 8 threads and now i can do 8 GPU rendering. I know It was more logical to get the i7-8700k that has 6 cores with 12 threads for 12 GPUs but the mobo I have doesn't support the new Coffee Lake processors. Poor me. But thanks for everyone's input!

 

EDIT: 

Also, if you're interested in speeds, I compared it with a GTX 1080 with the same scene

-1 GTX 1080 with 2560 CUDA cores. It took 1 HR 30 Mins to reach 2400 interations

-8 GTX 1080 ti's with 3584 CUDA cores each for a total of 28,672 CUDA cores it took 17 minutes to reach 2400 interations

Edit 2:

The whole time i was actually rendering with 5 GTX 1080 ti's. Not sure what happened but yeah...with 5 GTX 1080 ti took 17 minutes

WendyLuvsCatz

I had thought later you could test using less threads if it used less GPU's

well you can still use all 12 to cryptomine still I guess 

then buy the rest of the parts needed to build another PC from your spares for rendering on too

Padone

I believe windows 10 itself has a limit of handling max 8 gpus .. though it seems there's a workaround for ATI cards ..

https://answers.microsoft.com/en-us/windows/forum/windows_10-hardware/windows-10-have-a-limit-to-recognize-12-gpu/4ba16de2-3091-40f1-9d3d-acd3a5dead1d

http://www.aurelp.com/2017/11/01/solved-how-to-add-more-than-8-gpus-in-windows-10-step-by-stepfix/

 

this is also interesting

https://www.youtube.com/watch?v=jMjyZl3ri1E

https://www.youtube.com/watch?v=Z3i6oO7Gv0Q

 

 

xdeome

They already fixed that with recent windows updates. I already have the 12 GPUs recognized by windows in the device manager, so there is no issue with that. My original issue was that i couldnt render with more than 4 because Daz keeps throwing errors about not enough CPU threads. But I upgraded to a 8 Thread CPU and now I can render with 8 GPUs.

Since then, I have been running into a few issues though. With larger scenes around 5 GB VRAM+ i get errors saying that there wasn't enough memory to allocate on the cards so it doesn't render with them which is strange, because I have plenty of VRAM left as indicated on GPU-Z and other GPU monitoring programs. So sometimes DAZ gets a fatal error crash, or sometimes it just proceeds to render with fewer cards. With smaller scenes, it renders fine with 8 GPUs.

beowulfkaine_f87c8245ee

Any updates on speeds and rendering?

JD_Mortal

Have you tried setting them all to TCC mode? (Except the one used for your display, which needs to be in WDDM mode.)

https://docs.nvidia.com/gameworks/content/developertools/desktop/nsight/tesla_compute_cluster.htm

They will just be "hardware" in TCC mode, not "video display cards". Windows will not reserve VRAM from those cards and thus, also not have any "core needs" for them. (WDDM uses all available VRAM as one large chunk of virtual video ram.)

That CPU has only 4 single-thread cores... (4 physical/logical cores)... Daz normally only uses about 4 cores normally, even on my i7-7980xe 18/36 physical/logical core CPU. However, I only have 4 video cards running.

I would suggest trying to set half your cards into TCC mode, or all but four, and see if that helps. (The number of threads it may be talking about might actually be "CPU Program threads", not "CPU Processor threads". They are not the same thing.) However, unless you play games which windows may get use from shared VRAM, I would set them all into TCC mode, except the main card you use for display.

Do you have another setup that you can drop half those cards into? (Run 4 normal and 4 in TCC mode, on each machine. You can use the render to file setting to render on the remote machine, or get the cluster-render program.) I assume you are also using this setup as part of a beowulf cluster? (Lucky guess)

P.S. Daz is having non-specific memory issues in these latest 2 versions (4.11 and 4.12)... So there may actually be NOTHING wrong with your setup at all. Windows reports up to 80,000 program threads being used when I run Daz3D, which is a bit high, but not honestly abnormal to most operations. However, Daz can also run wild and consume up to 62GB out of my 64GB of RAM, for no apparent reason, in only a 2GB scene. Pushing the limits of my RAM and windows itself. Normally crashing after that point, or randomly way before reaching that point. (Similar with GPU and VRAM, but that is a roll of the dice, per scene, per use.)

kenshaw011267

1080ti's cannot be put into TCC. Only Titans and quadros can (Tesla's are always in TCC since they have no video outs).

BTW this thread is 18 months old. Let it die.

panton41_dfeb1322a2

I'm honestly surprised consumer Windows (Home and Pro) can handle more than, say, 3-4 discrete graphics cards. Enterprise or Windows Server, sure, because users pay through the nose annually for the benefit, but consumer Windows tends to be limited to more mainstream setups. (Though I seem to recall there being a Windows 10 HECP/Workstation version, but I could be mistaken.)