.. Or try to rotate IBL image (there is a button for that in IBL transform section) until the light is not reflecting anymore.
Note: IBL transform like any other can be animated on the timeline.
Just do not forget to click the button again after you are done with IBL rotation.
It's weird, but you will not be able to rotate anything else until you click it OFF.