No you did not imagine it. You possibly clicked Rotate IBL at some point. But never clicked it again to set IBL rotation OFF.
So when you think you are rotating a Directional Light and it is not moving, you are actually rotating IBL. A little weird. But that how it works.
You have to click out Rotate IBL to be able to rotate Directional light.
Edit: yes, that is if it's visible as Animagic suggested :)