Yes, if only tracking wheel encoders were used do calculate position of the robot that would be true.
However, if sensor fusion algorithm is used and knows about multiple parameters, such as control commands sent to the motors, readings from the powered wheels’ built-in encoders, any ultrasonic and line following sensors, etc… then it will be able to tell the difference.
Just using powered wheel encoders you can roughly tell where you are. The errors come from the slop in the drivetrain and powered wheels potentially slipping. With tracking wheels it will be able to detect and mostly resolve those errors.
If the rest of your robot needs more sensors you can attach only two tracking wheels and invest into studying math associated with using Kalman filter.
Alternatively, you can install three tracking wheels and get creative with attaching multiple sensors to the two remaining legacy ports. Having limit switches connected in parallel is not the only option.