Inertial sensor switching direction

These quotes give some insights on the way Inertial Sensor processes measurements:

Also, IMU diagnostics example: V5 internal sensor Code - #3 by jpearman

It is hard to say what is going on without more detailed debugging information. I would start by opening IMU dashboard and observing all available numbers as you manually rotate the robot.