Iray and HDRI Lighting

Wyrmfoe

I've noticed that when I use scene lighting, I can render a scene much faster than I can when using the environment lighting from an HDRI background.  I've also noticed that the render time gets worse when I am using a prim as a reflector.  What is the explanation for this?  Is it because the renderer is having to map each pixel each ray of light hits and with an HDRI, the light is coming from everywhere rather than with a mesh or photometric light where the light comes from a single origin, has steeper falloff, and generally less strength?