My experience running 3 graphics cards

areg5

As some of you may know, I’ve have been running three EVGA GTX 1080 Ti’s for a couple of years now.  From the onset, I have been unable to have all of them run at 100% power, and have had to throttle them back for them all to run to completion of the scene.  My first rig was an air cooled i7 4790 gigabyte GA z-97x Gaming 5, 32 gig Ram, 1200 W, Coolermaster Haf X.  I wasn’t sure what the issue was, and none of the suggestions I tried that I heard in this forum worked.  I figured that maybe the MB wasn’t up to it, or the processor for that matter.  I was using EVGA Precision XOC to fine tune the cards. I thought maybe a new build, with a more robust MB and AMD threadripper might allow all of the cards to run at full power.

So My latest build is an AMD Ryzen Threadripper 2950X, ASRock Fatality X399, 64 gig ram, Thermaltake Floriing, 1300 W, Thermaltake View71 TG case..  I was running the cards using the update to XOC, Precision X1.  The bottom line: I still could not get all three cards to run unless I throttled back one of the cards almost to the extent that it was only used for a few minutes on a render.

I had resigned to basically a 2 card setup, but even on that I had to throttle back one of the cards. I posed the question in the Tom’s Hardware forum.  I got an answer, and I think… crossing my fingers… I think the issue is solved. Many, many thanks to jrlaudio in that forum for his insight and knowledge.  He has been running a three card system for a while… three Titans!  I don’t know what he’s using them for, but I want that system.  Anyway, he helped me dispel some rumors.

Rumor 1: temperature.  I was concerned that one of the cards was running hot, almost into the 80’s.  He said, that’s not an issue.  A 1080 Ti can safely run into the low 90’s.

Rumor 2: EVGA X1 or XOC.  Ok.  My cards are all EVGA.  It only makes sense that I would manage them using EVGA software.  These cards are great.  I always get EVGA because I typically get them used on Ebay, and the warranty is for 3 years and is transferrable.  However, jrlaudio told me that those products, especially X1, are specifically designed to manage RTX cards and are not backwards compatible, and are notoriously bad when using Daz.  He recommended MSI Afterburner.  I tried it and saw an immediate difference.  I was able to run 2 cards at 100%.  The only thing I did with afterburner was adjust the fan curve.  My fans are all off until the card hits 50C, then they are all at 100%. If you are running more than one card, do not use XOC or X1 under any circumstances regardless of the make of the card.  Use MSI Afterburner.

Rumor 3: I might be underpowered for running 3 cards but certainly not for 2.  The jury is still out on that.

Rumor 4: Overclocking the cards, and SLI.  These are 2 things that should not be done in renders.  Overclocking leads to instability, and the cards can quit.  SLI essentially causes the linked cards to behave like a single card, meaning that it will cut the number of usable CUDA cores in half and slow the renders.

Now, when I built the new rig, I chose a MB with 4 slots in case I ever wanted to run 4 cards. If any of you have ever tried to build a system with three cards, with a standard setup 2 of the cards end up on top of each other.  They are almost touching.  In fact, I had to put a little wad of paper between them so they wouldn’t rattle on each other under load. Since these cards are so physically close to one another, it is inevitable that one or both of them will run very hot.  I did a few renders with this configuration, and the typical scenario would be one card ran in the 60’s, one in the 70’s, and then the hot one in the low 90’s.  I thought the temp differences might be a factor in why these renders were quitting. As the temp goes up, the cards demand more power. My power supply is a 1300 W gold, and from what I have read, the golds are ok for gaming but for serious work like rendering all day and all night maybe not so much, as they do not deliver that amount of power consistently, but in bursts. Could that be enough to quit to CPU?  Maybe.  Maybe it is during one of these random bursts that the cards quit.

Further research revealed that one of the cards could actually be mounted vertically, and my case actually came with a bracket to do just that. All I needed was a PCI-E cable to connect to it, so I got one. Then the cards would not be physically so close to one another.

I mounted the third card vertically. Huge, huge difference.  All cards now run at 100%, and they all run much cooler.  One of the cards gets up to 76 or 77 on a long render, the others stay in the 50’s.  I did a small batch render today, one of them took 30 minutes, and the other was a long one and ran for more than 2 hours.  But the main issue is that the system did not quit.

So in summary, the take home points for a 3 card build:

1. Do not use Precision XOC or X1.  They do not work well with Daz.  Use MSI afterburner.  Do not try to overclock the cards. Do not run them in SLI. The only adjustment to make should be on the fan curves.
2. Packing is a major issue in a three card rig.  Most MB’s will allow you 2 cards with plenty of space between them, but as soon as you go to three, you need to somehow mount the third card where it can breathe better.  I am really happy with the vertical mount.  You need a big case for it, and if it is a new build and you are planning three cards, choose one that either has a vertical mount bracket for one of the cards, or at the very least will be big enough to allow an aftermarket bracket installation. The other solution for cards on top of each other may be water cooling, but although I have a FloRiing on the AMD chip, I don’t have firsthand experience with hybrid or water cooled cards so I can’t really comment on that.
3. Power.  Like I said, the jury is still out.  I already pulled the trigger and ordered a Corsair 1600 W titanium.  Pay attention to whether the PSU is Gold, Platinum or Titanium.  They are not just names.  They reflect how consistent the PSU’s are in delivering power.  If you are serious about doing this, I would go with either a platinum or titanium, and I would always get more power than I think I need.

TheMysteryIsThePoint

2950X. Wow. I run a 1950 and it is ridiculously fast for compiling. I probably don't do anything that would even challenge a 2950X at all.

Anyway, detailed experience like yours based on actual experience rather than theory is always welcome.

I would add:

1) Get blowers. I fell asleep once and I burned my calf on the side of my case with 2 1080tis. Now, with 4 2080tis, the case is actually cooler but the air coming out the back would pop popcorn. And like you said, three of the four are mounted with MINIMAL space between them, but it doesn't seem to matter.

2) Risers don't seem to affect performance at all.

3) The efficiency rating of the PSU does indeed matter. I hadn't thought of what you said, but I know my electric bill went down by $20 when I got the 1600 Platinum (Of course it had gone up by $120). Makes me wonder if I shouldn't have gotten a Titanium, like you suggested you might.

4) Fewer people run these kinds of setups, and it seems like vendors don't test their products as well for them. You might be on your own if something doesn't work like the vendor said it would. I spent weeks getting my Aorus Gaming 7 to work with all four cards, but I never did get my SATA ports back, and the two RTX Titans I returned never did produce a video signal. I'm not sure my next system will be a homebuilt, I'm that traumatized.

kenshaw011267

Vertical GPU mounts are very case dependent. Most cases put the card all the way up against the glass and the fans can't move any air. The only time I'd do a vertical mount is in a custom loop.

No, the ratings have nothing to do with how consitent the power delivery is. They just say the minimum efficiency the PSU guarantees. Basically any decent name brand PSU will delivery consistent clean power unless it is running at 100% load, all PSU's get weird at full load. It is inherent in how they are designed. The higher rating will mean you draw less power which is a very good thing if you're rendering a lot. For instance a 1600W platinum PSU supplying 800W to the system would actually be drawing 870W while a Titanium would draw 851W. That excess power is wasted by the PSU as heat but it does increase your power bill and your cooling bill .

A 1200W PSU may have been one reason your first system was struggling. 3 base 1080ti would draw somewhere near 750W. The 4790 is another 85W. Plus drives, fans and the motherboard itself would be pushing 1000W minimum. If any of the EVGA cards was factory overclocked that would increase the load a lot.

The Haf X was a good case for the time, and it did move a lot of air but for 3 cards? There's just not enough airflow for that. You likely needed a bigger case. The View 71 TG is a good choice for 2 cards but for 3 it might have been a little small as well.

For motherboards a standard ATX board is going to be a tight fit for 3 aircooled cards. I'd go for an eATX card with one more PCIE slot to get some more spacing between the cards.

TBH trying to get a 3 aircooled card rig working in a standard case is never likely to work that well. For those kinds of setups custom loops are the way to go or build the whole thing on a testbench and use a deskfan to move lots of air across it or build it in a server case with server fans blowing air across the cards.

areg5

TheMysteryIsThePoint said:

2950X. Wow. I run a 1950 and it is ridiculously fast for compiling. I probably don't do anything that would even challenge a 2950X at all.

Anyway, detailed experience like yours based on actual experience rather than theory is always welcome.

I would add:

1) Get blowers. I fell asleep once and I burned my calf on the side of my case with 2 1080tis. Now, with 4 2080tis, the case is actually cooler but the air coming out the back would pop popcorn. And like you said, three of the four are mounted with MINIMAL space between them, but it doesn't seem to matter.

2) Risers don't seem to affect performance at all.

3) The efficiency rating of the PSU does indeed matter. I hadn't thought of what you said, but I know my electric bill went down by $20 when I got the 1600 Platinum (Of course it had gone up by $120). Makes me wonder if I shouldn't have gotten a Titanium, like you suggested you might.

4) Fewer people run these kinds of setups, and it seems like vendors don't test their products as well for them. You might be on your own if something doesn't work like the vendor said it would. I spent weeks getting my Aorus Gaming 7 to work with all four cards, but I never did get my SATA ports back, and the two RTX Titans I returned never did produce a video signal. I'm not sure my next system will be a homebuilt, I'm that traumatized.

That's understandable. You returned 2 ZRTX Titans?  That totally sucks.

areg5

kenshaw011267 said:

Vertical GPU mounts are very case dependent. Most cases put the card all the way up against the glass and the fans can't move any air. The only time I'd do a vertical mount is in a custom loop.

No, the ratings have nothing to do with how consitent the power delivery is. They just say the minimum efficiency the PSU guarantees. Basically any decent name brand PSU will delivery consistent clean power unless it is running at 100% load, all PSU's get weird at full load. It is inherent in how they are designed. The higher rating will mean you draw less power which is a very good thing if you're rendering a lot. For instance a 1600W platinum PSU supplying 800W to the system would actually be drawing 870W while a Titanium would draw 851W. That excess power is wasted by the PSU as heat but it does increase your power bill and your cooling bill .

A 1200W PSU may have been one reason your first system was struggling. 3 base 1080ti would draw somewhere near 750W. The 4790 is another 85W. Plus drives, fans and the motherboard itself would be pushing 1000W minimum. If any of the EVGA cards was factory overclocked that would increase the load a lot.

The Haf X was a good case for the time, and it did move a lot of air but for 3 cards? There's just not enough airflow for that. You likely needed a bigger case. The View 71 TG is a good choice for 2 cards but for 3 it might have been a little small as well.

For motherboards a standard ATX board is going to be a tight fit for 3 aircooled cards. I'd go for an eATX card with one more PCIE slot to get some more spacing between the cards.

TBH trying to get a 3 aircooled card rig working in a standard case is never likely to work that well. For those kinds of setups custom loops are the way to go or build the whole thing on a testbench and use a deskfan to move lots of air across it or build it in a server case with server fans blowing air across the cards.

Using a server setup was suggested to me at Tom's Hardware, but the cost is higher.  Yes, that might have been the way for me to go.  But the issue with running thee cards, one of the issues anyway, is definiftely how they are physically packed.And yes, 1200W in my first rig may have been too little power.  I think 1300 W is on the borderline, but so far has seemed to work, with this configuration anyway.

robertswww

Thanks for sharing your multi-card set-up experiences... some very helpful info in this thread!

TheMysteryIsThePoint

areg5 said:

That's understandable. You returned 2 ZRTX Titans?  That totally sucks.

@areg5 "Sucks" does not even begin to describe it... But ultimately I think 4 2080s is the better setup. When I get over my irrational fear to even open my case again :) , I'll try bridging two of them. 22 gigs of VRAM would almost completely cancel the benefit of the Titans.

I wonder if one could use a riser with the bridge, so I could have two sets of bridged cards? I wonder if the driver would even support that?

areg5

Bridging them?  What do you mean by that?  SLI? That doesn't double the VRAM.

ParallaxCreates

areg5 said:

As some of you may know, I’ve have been running three EVGA GTX 1080 Ti’s for a couple of years now.  From the onset, I have been unable to have all of them run at 100% power, and have had to throttle them back for them all to run to completion of the scene.  My first rig was an air cooled i7 4790 gigabyte GA z-97x Gaming 5, 32 gig Ram, 1200 W, Coolermaster Haf X.  I wasn’t sure what the issue was, and none of the suggestions I tried that I heard in this forum worked.  I figured that maybe the MB wasn’t up to it, or the processor for that matter.  I was using EVGA Precision XOC to fine tune the cards. I thought maybe a new build, with a more robust MB and AMD threadripper might allow all of the cards to run at full power.

So My latest build is an AMD Ryzen Threadripper 2950X, ASRock Fatality X399, 64 gig ram, Thermaltake Floriing, 1300 W, Thermaltake View71 TG case..  I was running the cards using the update to XOC, Precision X1.  The bottom line: I still could not get all three cards to run unless I throttled back one of the cards almost to the extent that it was only used for a few minutes on a render.

I had resigned to basically a 2 card setup, but even on that I had to throttle back one of the cards. I posed the question in the Tom’s Hardware forum.  I got an answer, and I think… crossing my fingers… I think the issue is solved. Many, many thanks to jrlaudio in that forum for his insight and knowledge.  He has been running a three card system for a while… three Titans!  I don’t know what he’s using them for, but I want that system.  Anyway, he helped me dispel some rumors.

Rumor 1: temperature.  I was concerned that one of the cards was running hot, almost into the 80’s.  He said, that’s not an issue.  A 1080 Ti can safely run into the low 90’s.

Rumor 2: EVGA X1 or XOC.  Ok.  My cards are all EVGA.  It only makes sense that I would manage them using EVGA software.  These cards are great.  I always get EVGA because I typically get them used on Ebay, and the warranty is for 3 years and is transferrable.  However, jrlaudio told me that those products, especially X1, are specifically designed to manage RTX cards and are not backwards compatible, and are notoriously bad when using Daz.  He recommended MSI Afterburner.  I tried it and saw an immediate difference.  I was able to run 2 cards at 100%.  The only thing I did with afterburner was adjust the fan curve.  My fans are all off until the card hits 50C, then they are all at 100%. If you are running more than one card, do not use XOC or X1 under any circumstances regardless of the make of the card.  Use MSI Afterburner.

Rumor 3: I might be underpowered for running 3 cards but certainly not for 2.  The jury is still out on that.

Rumor 4: Overclocking the cards, and SLI.  These are 2 things that should not be done in renders.  Overclocking leads to instability, and the cards can quit.  SLI essentially causes the linked cards to behave like a single card, meaning that it will cut the number of usable CUDA cores in half and slow the renders.

Now, when I built the new rig, I chose a MB with 4 slots in case I ever wanted to run 4 cards. If any of you have ever tried to build a system with three cards, with a standard setup 2 of the cards end up on top of each other.  They are almost touching.  In fact, I had to put a little wad of paper between them so they wouldn’t rattle on each other under load. Since these cards are so physically close to one another, it is inevitable that one or both of them will run very hot.  I did a few renders with this configuration, and the typical scenario would be one card ran in the 60’s, one in the 70’s, and then the hot one in the low 90’s.  I thought the temp differences might be a factor in why these renders were quitting. As the temp goes up, the cards demand more power. My power supply is a 1300 W gold, and from what I have read, the golds are ok for gaming but for serious work like rendering all day and all night maybe not so much, as they do not deliver that amount of power consistently, but in bursts. Could that be enough to quit to CPU?  Maybe.  Maybe it is during one of these random bursts that the cards quit.

Further research revealed that one of the cards could actually be mounted vertically, and my case actually came with a bracket to do just that. All I needed was a PCI-E cable to connect to it, so I got one. Then the cards would not be physically so close to one another.

I mounted the third card vertically. Huge, huge difference.  All cards now run at 100%, and they all run much cooler.  One of the cards gets up to 76 or 77 on a long render, the others stay in the 50’s.  I did a small batch render today, one of them took 30 minutes, and the other was a long one and ran for more than 2 hours.  But the main issue is that the system did not quit.

So in summary, the take home points for a 3 card build:

1. Do not use Precision XOC or X1.  They do not work well with Daz.  Use MSI afterburner.  Do not try to overclock the cards. Do not run them in SLI. The only adjustment to make should be on the fan curves.
2. Packing is a major issue in a three card rig.  Most MB’s will allow you 2 cards with plenty of space between them, but as soon as you go to three, you need to somehow mount the third card where it can breathe better.  I am really happy with the vertical mount.  You need a big case for it, and if it is a new build and you are planning three cards, choose one that either has a vertical mount bracket for one of the cards, or at the very least will be big enough to allow an aftermarket bracket installation. The other solution for cards on top of each other may be water cooling, but although I have a FloRiing on the AMD chip, I don’t have firsthand experience with hybrid or water cooled cards so I can’t really comment on that.
3. Power.  Like I said, the jury is still out.  I already pulled the trigger and ordered a Corsair 1600 W titanium.  Pay attention to whether the PSU is Gold, Platinum or Titanium.  They are not just names.  They reflect how consistent the PSU’s are in delivering power.  If you are serious about doing this, I would go with either a platinum or titanium, and I would always get more power than I think I need.

Wow, what an amazing read. I can't thank you enough for sharing this as I know it took some time to put together. I have always used Afterburner up until I started running 1080tis. The only reason I made the switch over to X1 is because I wasn't digging the fact that 2nd fan wasn't kicking in until later and with X1 I can set the fan curve for it. I'm going to update my Afterburner and run some tests.

areg5

You're welcome. It has been a long struggle and I have learned a lot, and am still learning a lot, so I thought I would share my real world experience with all of this. My current set up still can occasionally hang for reasons that are very unclear.  It will run a 6 character render but hang on a two character render.  WTF?! In those cases I have to throttle down again for that scene, but then I can throttle up for the next, and they run.  I bet when I get the Titanium PSU it will run no problems. Heh.  Famous last words!

But yeah, avoid the Precision X1 like the plague. It does not give accurate information.  I noticed that with Afterburner, the fans that I can see (since the vertical mounting) both turn when it gets hot enough, but early in the fan curve only one of them runs. 

ParallaxCreates

areg5 said:

You're welcome. It has been a long struggle and I have learned a lot, and am still learning a lot, so I thought I would share my real world experience with all of this. My current set up still can occasionally hang for reasons that are very unclear.  It will run a 6 character render but hang on a two character render.  WTF?! In those cases I have to throttle down again for that scene, but then I can throttle up for the next, and they run.  I bet when I get the Titanium PSU it will run no problems. Heh.  Famous last words!

But yeah, avoid the Precision X1 like the plague. It does not give accurate information.  I noticed that with Afterburner, the fans that I can see (since the vertical mounting) both turn when it gets hot enough, but early in the fan curve only one of them runs. 

I now run two EVGA 1080 Tis SC2 and just ordered a new psu as I am sure after reading your post my EVGA 850 G2 isn't enough. Thanks again.

areg5

ParallaxCreates said:

areg5 said:

You're welcome. It has been a long struggle and I have learned a lot, and am still learning a lot, so I thought I would share my real world experience with all of this. My current set up still can occasionally hang for reasons that are very unclear.  It will run a 6 character render but hang on a two character render.  WTF?! In those cases I have to throttle down again for that scene, but then I can throttle up for the next, and they run.  I bet when I get the Titanium PSU it will run no problems. Heh.  Famous last words!

But yeah, avoid the Precision X1 like the plague. It does not give accurate information.  I noticed that with Afterburner, the fans that I can see (since the vertical mounting) both turn when it gets hot enough, but early in the fan curve only one of them runs. 

I now run two EVGA 1080 Tis SC2 and just ordered a new psu as I am sure after reading your post my EVGA 850 G2 isn't enough. Thanks again.

Sounds like a good plan! I've done many a render with those 2 cards.  They're great!

 

TheMysteryIsThePoint

areg5 said:

Bridging them?  What do you mean by that?  SLI? That doesn't double the VRAM.

I render in Blender Cycles on Linux. The NVidia driver supports NVLink on the 2080s, and is supposed to do VRAM pooling, if I understood correctly. I was doubly unsure if it would work for more than 2 GPUs, or if I could bridge just two. I know there are 4-way bridges, but not all 4 of the 2080s are adjacent to each other; one is mounted on the case.

 

kenshaw011267

TheMysteryIsThePoint said:

areg5 said:

Bridging them?  What do you mean by that?  SLI? That doesn't double the VRAM.

I render in Blender Cycles on Linux. The NVidia driver supports NVLink on the 2080s, and is supposed to do VRAM pooling, if I understood correctly. I was doubly unsure if it would work for more than 2 GPUs, or if I could bridge just two. I know there are 4-way bridges, but not all 4 of the 2080s are adjacent to each other; one is mounted on the case.

You should be able to have 2 NVLink bridges in the same system. The two pairs will operate independently. I'm unaware of a 4 way NVLink bridge. There are 4 slot wide bridges but those only have 2 connectors.

 

kenshaw011267

areg5 said:

kenshaw011267 said:

Vertical GPU mounts are very case dependent. Most cases put the card all the way up against the glass and the fans can't move any air. The only time I'd do a vertical mount is in a custom loop.

No, the ratings have nothing to do with how consitent the power delivery is. They just say the minimum efficiency the PSU guarantees. Basically any decent name brand PSU will delivery consistent clean power unless it is running at 100% load, all PSU's get weird at full load. It is inherent in how they are designed. The higher rating will mean you draw less power which is a very good thing if you're rendering a lot. For instance a 1600W platinum PSU supplying 800W to the system would actually be drawing 870W while a Titanium would draw 851W. That excess power is wasted by the PSU as heat but it does increase your power bill and your cooling bill .

A 1200W PSU may have been one reason your first system was struggling. 3 base 1080ti would draw somewhere near 750W. The 4790 is another 85W. Plus drives, fans and the motherboard itself would be pushing 1000W minimum. If any of the EVGA cards was factory overclocked that would increase the load a lot.

The Haf X was a good case for the time, and it did move a lot of air but for 3 cards? There's just not enough airflow for that. You likely needed a bigger case. The View 71 TG is a good choice for 2 cards but for 3 it might have been a little small as well.

For motherboards a standard ATX board is going to be a tight fit for 3 aircooled cards. I'd go for an eATX card with one more PCIE slot to get some more spacing between the cards.

TBH trying to get a 3 aircooled card rig working in a standard case is never likely to work that well. For those kinds of setups custom loops are the way to go or build the whole thing on a testbench and use a deskfan to move lots of air across it or build it in a server case with server fans blowing air across the cards.

Using a server setup was suggested to me at Tom's Hardware, but the cost is higher.  Yes, that might have been the way for me to go.  But the issue with running thee cards, one of the issues anyway, is definiftely how they are physically packed.And yes, 1200W in my first rig may have been too little power.  I think 1300 W is on the borderline, but so far has seemed to work, with this configuration anyway.

Buying a new server case is expensive. You can get a used one for peanuts. You'd need to mod in standoffs for your motherboard bbut plenty of people have done that I'm sure you can find how to's online.

The reason a server case would improve airflow is these:

https://www.ebay.com/c/1877794828

4 of those, or 8 in really big cases, moves an awful lot of air.

areg5

kenshaw011267 said:

areg5 said:

kenshaw011267 said:

Vertical GPU mounts are very case dependent. Most cases put the card all the way up against the glass and the fans can't move any air. The only time I'd do a vertical mount is in a custom loop.

No, the ratings have nothing to do with how consitent the power delivery is. They just say the minimum efficiency the PSU guarantees. Basically any decent name brand PSU will delivery consistent clean power unless it is running at 100% load, all PSU's get weird at full load. It is inherent in how they are designed. The higher rating will mean you draw less power which is a very good thing if you're rendering a lot. For instance a 1600W platinum PSU supplying 800W to the system would actually be drawing 870W while a Titanium would draw 851W. That excess power is wasted by the PSU as heat but it does increase your power bill and your cooling bill .

A 1200W PSU may have been one reason your first system was struggling. 3 base 1080ti would draw somewhere near 750W. The 4790 is another 85W. Plus drives, fans and the motherboard itself would be pushing 1000W minimum. If any of the EVGA cards was factory overclocked that would increase the load a lot.

The Haf X was a good case for the time, and it did move a lot of air but for 3 cards? There's just not enough airflow for that. You likely needed a bigger case. The View 71 TG is a good choice for 2 cards but for 3 it might have been a little small as well.

For motherboards a standard ATX board is going to be a tight fit for 3 aircooled cards. I'd go for an eATX card with one more PCIE slot to get some more spacing between the cards.

TBH trying to get a 3 aircooled card rig working in a standard case is never likely to work that well. For those kinds of setups custom loops are the way to go or build the whole thing on a testbench and use a deskfan to move lots of air across it or build it in a server case with server fans blowing air across the cards.

Using a server setup was suggested to me at Tom's Hardware, but the cost is higher.  Yes, that might have been the way for me to go.  But the issue with running thee cards, one of the issues anyway, is definiftely how they are physically packed.And yes, 1200W in my first rig may have been too little power.  I think 1300 W is on the borderline, but so far has seemed to work, with this configuration anyway.

Buying a new server case is expensive. You can get a used one for peanuts. You'd need to mod in standoffs for your motherboard bbut plenty of people have done that I'm sure you can find how to's online.

The reason a server case would improve airflow is these:

https://www.ebay.com/c/1877794828

4 of those, or 8 in really big cases, moves an awful lot of air.

Well, I like my case.  In fact, I have room for two 120's in the base and I think two on top. I have three in front, and one in the back.  Now I did stall on a couple of scenes, like two out of 20 or so. It stopped happening when I left the door open. Temps went way down. And no more stalls. So when I replace the psu, I'll put in as many fans as is possible. 

kenshaw011267

No reason to not just leave the side off, unlesyou have kids or pets. Putting lots of fans in is tricky. Make sure to have more intake than exhaust so you'll have a positive pressure system and get less dust in the rig.

Former DAZ Customer

The "packing them in" issue is why I opted for Founders Edition (blower) cards in my 3-GPU setup.  Renders run at 100% and temps stay under control with Precision X1 (current builds work fine with GTX10## cards, in my experience).

It might be worthwhile to replace in-case fan coolers with water blocks to keep temps under control.

areg5

Gordon Runkle said:

The "packing them in" issue is why I opted for Founders Edition (blower) cards in my 3-GPU setup.  Renders run at 100% and temps stay under control with Precision X1 (current builds work fine with GTX10## cards, in my experience).

It might be worthwhile to replace in-case fan coolers with water blocks to keep temps under control.

 

Gordon Runkle said:

The "packing them in" issue is why I opted for Founders Edition (blower) cards in my 3-GPU setup.  Renders run at 100% and temps stay under control with Precision X1 (current builds work fine with GTX10## cards, in my experience).

It might be worthwhile to replace in-case fan coolers with water blocks to keep temps under control.

Maybe the next one will be watercooled. That's not a bad idea, going with Founder's Editions for packing.  When you say the temps are under control, hot how does it get? Also, what is your PSU? Are your cards all 1080 Ti's? My build has 2 SC's and a Black Edition. Although they are similar, they do have unique features, like the difference in burst speed and LED's.  I have wondered if that is enough to cause instability in my case.  If X1 works for you, I won't say don't use it.  But for me, I had to set X1 to 80% on one of the cards just to have TWO of them run all the way through.  Afterburner eliminated that need for me, and will run all 3 cards to completion more times than not.

areg5

kenshaw011267 said:

No reason to not just leave the side off, unlesyou have kids or pets. Putting lots of fans in is tricky. Make sure to have more intake than exhaust so you'll have a positive pressure system and get less dust in the rig.

My plan is 2 base fans and one on top.  I have a cat, so I only leave the door open when I'm in the room with it.

ParallaxCreates

Btw, when you all are stating "100%", are you referring to usage or power? I don't think I have ever seen any of my GPUs reach 100% Power. This what my GPUs typically look like during a render. I'll be at pretty much 100% on usage but never above 80% on Power.

ParallaxCreates

Here's a better look at my system during a 20min render. 

areg5

I'm basically talking about the slider.  Where I set the power slider.  Now, this slider doesn't actually decrease the amount of power used. It limits the bosst, and when it's applied.  I think that's where the instability comes from with lesser PSU's, instability during the boost.

kenshaw011267

You might want to consider undervolting the cards. A mild undervolt will cost you relatively little, if any, frequency and you could save substantially on power consumption. Less heat and less power draw could solve a number of your issues.

Former DAZ Customer

Full on rendering, which HWMon shows as 100% GPU utilization on all 3 1080Ti FE gives me a max temp of 68-70C (the top and bottom run about 3-5C cooler) with X1 fan curves set.  HWMon reports the fans running around 80% speed.  Power supply is an EVGA G3 1200 Gold, motherboard is MSI X99A SLI Krait, CPU i7-5820K, case is CoolerMaster HAF 932 Advanced.

areg5

kenshaw011267 said:

You might want to consider undervolting the cards. A mild undervolt will cost you relatively little, if any, frequency and you could save substantially on power consumption. Less heat and less power

My guy at Tom's Hardware said not to do that under any circumstances for Iray rendering.  What he does, in fact, is up the max temps which ends up drawing more power.

areg5

Gordon Runkle said:

Full on rendering, which HWMon shows as 100% GPU utilization on all 3 1080Ti FE gives me a max temp of 68-70C (the top and bottom run about 3-5C cooler) with X1 fan curves set.  HWMon reports the fans running around 80% speed.  Power supply is an EVGA G3 1200 Gold, motherboard is MSI X99A SLI Krait, CPU i7-5820K, case is CoolerMaster HAF 932 Advanced.

Similar to my old rig, very much so with the exception of the processor. Maybe that was my problem.  Coolermaster is a good case.  I really do like it.  The Thermaltake is a bit airier. I've been leaving the door open today, and my hottest card is running at 67C.  The others are in the high 40's.

Bobvan

TheMysteryIsThePoint said:

2950X. Wow. I run a 1950 and it is ridiculously fast for compiling. I probably don't do anything that would even challenge a 2950X at all.

Anyway, detailed experience like yours based on actual experience rather than theory is always welcome.

I would add:

1) Get blowers. I fell asleep once and I burned my calf on the side of my case with 2 1080tis. Now, with 4 2080tis, the case is actually cooler but the air coming out the back would pop popcorn. And like you said, three of the four are mounted with MINIMAL space between them, but it doesn't seem to matter.

2) Risers don't seem to affect performance at all.

3) The efficiency rating of the PSU does indeed matter. I hadn't thought of what you said, but I know my electric bill went down by $20 when I got the 1600 Platinum (Of course it had gone up by $120). Makes me wonder if I shouldn't have gotten a Titanium, like you suggested you might.

4) Fewer people run these kinds of setups, and it seems like vendors don't test their products as well for them. You might be on your own if something doesn't work like the vendor said it would. I spent weeks getting my Aorus Gaming 7 to work with all four cards, but I never did get my SATA ports back, and the two RTX Titans I returned never did produce a video signal. I'm not sure my next system will be a homebuilt, I'm that traumatized.

Some of you guys are insane whats render times 30 seonds for a full indoor scene I hope :)

 

My advanced PC knowlege obviously sucks :)

areg5

Bobvan said:

TheMysteryIsThePoint said:

2950X. Wow. I run a 1950 and it is ridiculously fast for compiling. I probably don't do anything that would even challenge a 2950X at all.

Anyway, detailed experience like yours based on actual experience rather than theory is always welcome.

I would add:

1) Get blowers. I fell asleep once and I burned my calf on the side of my case with 2 1080tis. Now, with 4 2080tis, the case is actually cooler but the air coming out the back would pop popcorn. And like you said, three of the four are mounted with MINIMAL space between them, but it doesn't seem to matter.

2) Risers don't seem to affect performance at all.

3) The efficiency rating of the PSU does indeed matter. I hadn't thought of what you said, but I know my electric bill went down by $20 when I got the 1600 Platinum (Of course it had gone up by $120). Makes me wonder if I shouldn't have gotten a Titanium, like you suggested you might.

4) Fewer people run these kinds of setups, and it seems like vendors don't test their products as well for them. You might be on your own if something doesn't work like the vendor said it would. I spent weeks getting my Aorus Gaming 7 to work with all four cards, but I never did get my SATA ports back, and the two RTX Titans I returned never did produce a video signal. I'm not sure my next system will be a homebuilt, I'm that traumatized.

Some of you guys are insane whats render times 30 seonds for a full indoor scene I hope :)

 

My advanced PC knowlege obviously sucks :)

Nah.  Some complex renders still take a while.  My scenes tend to be very complex.  But I did do the Sickleyield Benchmark in 40 seconds, which sure ain't bad.

Bobvan

areg5 said:

Bobvan said:

TheMysteryIsThePoint said:

2950X. Wow. I run a 1950 and it is ridiculously fast for compiling. I probably don't do anything that would even challenge a 2950X at all.

Anyway, detailed experience like yours based on actual experience rather than theory is always welcome.

I would add:

1) Get blowers. I fell asleep once and I burned my calf on the side of my case with 2 1080tis. Now, with 4 2080tis, the case is actually cooler but the air coming out the back would pop popcorn. And like you said, three of the four are mounted with MINIMAL space between them, but it doesn't seem to matter.

2) Risers don't seem to affect performance at all.

3) The efficiency rating of the PSU does indeed matter. I hadn't thought of what you said, but I know my electric bill went down by $20 when I got the 1600 Platinum (Of course it had gone up by $120). Makes me wonder if I shouldn't have gotten a Titanium, like you suggested you might.

4) Fewer people run these kinds of setups, and it seems like vendors don't test their products as well for them. You might be on your own if something doesn't work like the vendor said it would. I spent weeks getting my Aorus Gaming 7 to work with all four cards, but I never did get my SATA ports back, and the two RTX Titans I returned never did produce a video signal. I'm not sure my next system will be a homebuilt, I'm that traumatized.

Some of you guys are insane whats render times 30 seonds for a full indoor scene I hope :)

 

My advanced PC knowlege obviously sucks :)

Nah.  Some complex renders still take a while.  My scenes tend to be very complex.  But I did do the Sickleyield Benchmark in 40 seconds, which sure ain't bad.

LOL I dont doubt it. I will have to rerun it once I slap in my 2080ti next week..

areg5

Nice.  I check Ebay every day for one.  When they start dropping under 1000, I'm there!