Rendering with Multiple Cores
Dartanbeck
Posts: 21,574
Rendering complex scenes with eight cores, I couldn't help but to notice that in an intense area, a single core could get bogged down - working all by itself on that vast portion. I figured that, if all of my cores could help with that spot, the whole process should work faster. So I cranked the rendering tile size down all the way - which is 16. This setting is at the very bottom of the rendering tab in the render room, under the network settings.
I haven't actually timed which is faster, but it only seems to make sense to me.
Daz 3D is part of 
Connect
DAZ Productions, Inc.
7533 S Center View Ct #4664
West Jordan, UT 84084
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
Absolutely! My final renders are nearly always done at 16 size bucket -- especially if hair or intricate shadows are involved. The whole idea of multiple cores is to divvy up a large amount of work into tiny pieces.
There is one major threading enhancement I'd like to see (if it is even possible). Once most of the threads reach the end of the image (which annoying reports 100% complete even if there are threads still working...), any threads that are still working re-evaluate their bucket and divide it up further.
For example, with the 16 size bucket, the bits you can actually see complete are 4x4 in size and there are four of them. Using numbers for threads, if thread 2 were working on an area and it was taking a long time (hair commonly) it would look like:
The asterisk marks the current 4x4 area being processed. It would be awesome if other threads could jump in to help. We'd end up with:
That's just a former code-jockey dreaming...there's plenty of smaller, easier tweaks that Carrara needs first...
HI Dartanbeck :)
It does depend on the area that's being rendered, .. Sometimes,,. in a simple scene, the Big Bucket size will cover an area of "relatively" empty scene space, quite quickly,. while it can take a little longer for smaller buckets (rendering blocks) to cover the same area.
BUT,.. the opposite also applies,. Big sized render buckets will take longer to work on a complex area.
It would be really nice it there was a flexible and automatic adjustment to the size of area being rendered.
For example,: If most of the image has been rendered, but one core is still working on a complex area, and there are some inactive cores available, which could be allocated to rendering the next available sections of that area.
in other words ,. Smart rendering where inactive cores, go back and help out the struggling core.
just a thought :)
Right...the tricky bit is in predicting those complex areas in advance.
Hmmm...caffeine is kicking in now...come to think of it, the problem is not all that difficult. Each thread captures a timestamp when it starts working on a bucket. After a certain period of time - even make that value configurable - it sends out a "Please help me!" signal that other idle threads can detect and react to.
As noted in my original reply, my biggest beef is that the progress reports 100% prematurely.
Yeah, and adaptive bucket sizing would be nice. There are certainly a few ways to detect how bad a given block will be - count up the objects in the zone, look for hair or plants, and do a probe of the shaders across the bucket looking for transparency (whether real or alpha). Give weights to them and add them up, and let us set the threshold at which it subdivides the buckets.
It would require some tweaking because the rendering system appears to allocate all of the buckets up front.
So... are you gonna make a new plugin for this?
LOL
C'mon you guys!
Carrara, as you know, is the fastest render monster we have.
As for those large areas of relatively blank scene... I love watching the blur! Those little 16x16 blocks whiz by so fast! But 3dage is probably right, as always... it likely takes slightly longer. But that's nothing compared to the time savings I think I'm getting from smaller tiles allowing all eight cores to solve every render issue.
I'm feeding each core 2 GB of RAM. Do you think it needs 4? My motherboard can handle 32 GB and I have 16. I couldn't afford it then, but I'll look at the upgrade. I bet it's pretty reasonable. But if you don't think it would help, I'll save the cash for more Daz products.
Say Fenric. What's the deal now. Is it against TOS for us to advertise your site here? Any plans on coming back to Daz, at least with a big bundle of all of your plugins?
I want each and every one of them. Aside from MDD exprot, I have all of your Daz offerings - and I'm going to sign in by you and buy all of yours too... but I'm working on that What is Carrara and How to Use It thread, which is intended to be like a user manual that lives. so you can ask and respond... etc.,
I wanted to do a write-up on you and link to your sight. To me, you and your tools are essential to Carrara. perhaps that's why I'm always happier than the average Carrara user? But I don't want to get in trouble with the admins - I am a Daz Artist, and want to keep that title for my whole life.
Come back?:grrr:
Another thing that we Carrara artists can do to help our cores is this:
Take a look at everything in the scene - everything.
Not all at once either. Analyze everything - and look for how YOU can help that render. Personally, I realized that it's really cool that this incredible texture set I bought for Rosie is one beautiful set of high resolution images and UV placement. I just would never want to try making these so nice - they put a LOT of work into optimizing these textures. For most purposes, I don't need to see the pores in her skin. I took all of her texture, specular, and bump maps into photoshop and reduced them to 500 x 500, which may be too small for some folks here, but you've all seen my Rosie renders - she looks great, I think. I try very hard to enhance models in ways that I can eliminate alpha. It seems to help. When I make a scene that focuses on something in the water, I naturally leave refraction ON in the render... otherwise I leave that off. I've also found that where I've been using reflections, what I really wanted was a nice blend in Highlight vs Shininess - which creates a very similar, yet much faster render.
Often times you really DO ned reflections - but many cases are as I've said.
I know you've all heard this before, but duplication and replication is essential if you want to keep resources down. But if you take the time to resize texture files from 4200 x 4200 down to 300 x 300, you'll save a lot of time, and 300 pixels on a small item int he scene is plenty - with a nice image file.
Just some thoughts for you to consider.
I can only advertise my site and non-DAZ brokered products in the "Commercial" sub-forum, not here. You can put whatever you want, wherever you want it, as long as you don't break any other rules :)
I'm not ruling out ever putting anything in the store here again, but it depends rather a lot on what DAZ announces this month (if they actually announce anything at all). I'm certainly not going to rely on them for distribution of updates ever again. If I put anything else here, it would be more like Inagoni where DAZ would have whatever version Spooky accepted through QA for initial downloading, but you would need to check my site for updates.
My nerves are rattled at the way you mention "The Announcement".
Damn... what a scary little phrase she dropped on us, eh? I sure hope it isn't a Bad news announcement.
All I can do now is to stay optimistic. And I am.
I was going to reply to your "Come visit my store!" before I realized that this is your sig. But I'm deciding to reply to it anyways:
I like your store. I like what you do, and I hope to see you continue - no matter Carraras future. Like Holly says, just because they stop changing it doesn't mean anyone has to stop using it. I know I surely won't. and if products take some new direction making them incompatible with Carrara, that would be the time I quit buying and call my purchased collection complete. I truly have had enough stuff to complete several types of movies for years now. I mainly collect them because I like to - it's fun opening a shiny new box. I never buy xbox games because I don't have an xbox console.
So what updates are you talking about? Do my Fenric plugins need updates?
You can also speed up rendering times by turning off unneeded render functions in the renderer. For instance, if I don't have anything with refraction, I'll disable that in the render settings. If I don't use the DOF, then I turn that off. Even if you don't have any of those functions in your scene, but leave the options enabled, Carrara still looks for those uneeded functions. I remember Sub7th (I wish he still posted here) said once he would disable the bump as well, unless it was absolutely needed.
Bucket size (in any rendering engine) is always a bit tricky. As 3DAge mentions, the most common way I determine what size I want to use is based on how many objects are in the scene. Scenes with many objects tend to render faster with smaller sizes. Especially if they have advanced surfaces with reflections, refraction, transparency, etc. The main issue with hair is that the fine grained tranparency maps are computationally expensive to render. On the other hand scenes with only a few simple objects in them are going to render much more quickly with large bucket sizes.
I've always wished that the engine could be smart enough to sub-divide rendering buckets once there were idle threads (cores) available. Pre-calculating bucket sizes for each section of a large image may be counter productive, but it would be nice if when 2 threads finish, they could look for other threads that are still working on their buckets and offer to take half of what's left there to render.
I wonder if Fenric could make something for that.
What do you think Fenric?
I would love to see a render optimization plugin from Fenric.
Yeah Fenric, could you teach our cores how to swarm the project at hand and efficiently churn out an even faster render?
I'm having a lot of fun imagining the looks washing over Fenric's face as he reads this thread... :-/ :shut: :sick: :ahhh:
Heh... I sure love that little fox! He just rock the world with code, doesn't he?
Darn it, Garstor... you have me watching little buckets as render now! Arghhhh!!!
lol
It is interesting how quickly electricity can fathom my imagination, isn't it? I'm barely able to follow dome of them, and some I can't at all. There's even some that I don't think I see at all! And I also like to turn to my progress panel before rendering. Did you ever see how many rays get launched within, like, half a second? Geez!
Ok... it's time to finally post this writeup for once. I've typed up my thoughts on the final renderer a dozen times, but never posted it.
Carrara is written in what I like to call "Academically Correct C++": it is PAINFULLY object-oriented.
If you aren't used to the concept, think of everything as a puppy that has learned some tricks: they don't always behave, but whatever they DO do, they do on their own. So in Carrara: things that can be rendered know a trick called "ShadeAndLight2". But even then, the item only knows so much - it depends on its shader to do much of the real work. "Top Level" shaders (surely you've seen that warning?) ALSO implement "ShadeAndLight2", and the renderable instance defers most if not all of its work to the shader.
So, the "final renderer" is little more than a manager: iterating over the available renderable objects, lights, backgrounds, atmospheres, and cameras - askeing each one to prepare its own little bit of the image and compositing the lot together. It does not actually perform the calculations that are the heart of the entire process.
But none of that is the part you are asking about. That is the very simple item called "I3DRenderHelper" that handles all threading and bucket (also known as "tile") management. It has an unfortunate line in the SDK documentation:
"You cannot implement your own RenderHelper."
Well that pretty much slams the vault door on the idea...
From what you described though, it sounds like DAZ could serious benefit from some hardcore code profiling and refactoring. Is it just me or did a cool breeze blow through my personal suite in Hades? :smirk:
Carrara does render pretty quick, at least.
Well, to be honest, except for a bit of bug fixing and making Carrara more friendly to premade Poser/DAZ Studio style content, Cararra hasn't changed a great deal since DAZ acquired it.
Has always been a bit frustrating for long term Carrara fans that there hasn't been more ongoing new development. We had the big change with dynamic hair and physics, but that's a few years ago now. Nothing even close to the yearly (or twice yearly) updates that most of Autodesk's products get.
On that happy note, I think I'll go download my LightWave 11.5 update. :coolsmile:
On that happy note, I think I'll go download my LightWave 11.5 update. :coolsmile:OMG is THAT ever funny!
I better go do something fun for a while. Yeah. Cuz' adding Dynamic hair is nothing to do. It's simple. Anyone can do it. But who Did it? Daz3d. Who rewrote everything to make this work in 64 bit? Oh but that's another easy task, isn't it? But who?
Daz3d.
God Rays, anyone? Or how about multi-pass rendering, UV unfolding, SubD enhancements in the vert modeler? Oh... and let's not forget the ability to use Normal Maps... you don't ever do that, do you? NLA is one of the sweetest things and animator can have. But that's no real improvement. How about being able to view OpenGL over 128x128, or the ability to select more than one item at a time? And now, I suppose all of this work towards compatibility that they've endlessly slaved over every day with fingers blazing across plastic keys is just all wasted time because they don't want to improve anything?
Drives me nuts when people bad-talk Pierre and his dedicated crew. NUTS! Those guys are the biggest reason Carrara still sells today.
Excuse me ... I have to go tear something apart with my bare hands
Didn't mean to zap that nerve with 50,000 volts Dart... :ohh:
I'm trying to use both Carrara and LightWave together. To be sure; there is one big thing Carrara does that I don't see (maybe its hidden) LW doing...exporting a UV map as PSD so you can create/edit textures.
Feel better? Here's some virtual hot cocoa with mini-marshmellows to take the edge of the Wisconsin winter... ;)
Sorry,
I shouldn't get so worked up over words that come from the same mouth that thinks, for some crazy notion, that Stefan Morel's use of displacement over polygons is somehow not right. We all know that there is a bit of Map work to do with Poser products in Carrara. Many products come equipped with displacement maps. they are very easy to tweak and work with! When Stonemason clips his pinky nail, more talent just fell into the garbage than...
"Sorry Rosie... you're absolutely right!"
Okay... I came to apologize for rude behavior, not embellish it. I'm sorry for blowing up. If you'll excuse me, I have to take my Wife to town now.
I'm glad you like making plugins. My Carrara is probably different that some others simply because of them. Edit > Fenric is a great habit to acquire, and something as simple as making a simple pz2 from within Carrara made my life so much easier. Just wanted to thank you for all of your work.
I think people just don't know. It would be helpful if you did mention their names from time to time as the regular ass-uming folks always speculate there aren't any Carrara developers. I know they are shared with DAZ's other work, but there are people who are very knowledgable about the program working on it.
Now I'm just confused...that ire couldn't have been (intentionally) directed at me because I have no idea what you are talking about now! :)
Enjoy the afternoon out...I think I'll take a quick outside break myself...
Are you kidding? You never ever bring the beast out in me, Garstor. We're bro's in crime... well... without the crime part! ;)
Well, we're not telling anyone about the crime part...are we? ARE WE?!?!?! ;-P
Now I have a vision of you hurling ornamental boulders around. Dartanbeck SMASH! :lol:
Hey, coding is tough work. I link to tinker with it, but every time I tried it "for real" I always turned out trash. Having once had viewing rights to the SQL Server source code; I am fully aware how spectacularly complicated major apps can be. Daz needs to do it twice...once for the PC and then again for the Mac.
Yeah, yeah...
I know these guys are tough! They really did some great stuff with Carrara since they've graced it with their expertise!
As for the boulders part, I really do move them by hand as much as possible to avoid scratching them. Many people wish they could lift like Dartanbeck without getting hurt - or dead! I am an animal on the job!
Last week, when I was wheeling concrete into that basement, the other guys though I was nuts when I filled my first wheelbarrow full. I did that all day long, often butting in front of them (mainly to give 'em a little break) at the concrete truck. I AM an animal at work! And I love it! Makes the veins stick out!
No need to apologize, but it does seem like maybe you're a bit confused about some of my recent posts / comments.
First, I don't think I've ever been critical of Stefan / Stonemason. I own pretty much everything he's ever made. I'm actually in awe of him and his skill. To be honest, I'm a bit confused about what comment I made which may have been interpreted of being critical of him and his products.
Second, my grief has never been with the developers at DAZ over anything. Even before most software became free, it's always been my standpoint that DAZ software provides the best value for the money by far. Carrara is a great example of this. The features it provides at the price point that DAZ makes it available is amazing.
My main thing when it comes to Carrara in particular hasn't been with the skills or abilities of the development team, it's been the level of focus and investment that DAZ as a company has been putting into the product. I just went and looked at my account history. I paid for my upgrade to 6.1 in 2008, I skipped version 7 because there wasn't enough in it to make me want to do the upgrade, then I went to 8.0 February of 2011.
That's 2 years without any significant changes. As Stan mentioned in another thread, 8.5 has been in beta now for 1.5 years or so. That shows a lack of focus at the corporate level on this piece of software. It has nothing to do with the abilities of the developers or the quality of what they produce, it's a matter of the level of investment in the package.