That may be the issue then. Encoder returns an integer as it is a digital sensor, meaning that the value is discrete and any decimal will be wiped. At some point the loop delay time will be too small that the decimals wiped becomes significant. For example, if you travel around 1.5 ticks every loop, it will get wiped into 1 ticks, meaning you have instantly lost 33% of distance u travelled. If the error continues 500 times per second, error will become very significant.
Even though lowering a loop time increases the accuracy due to having a better approximation, if it is too low, error from discretization will become too significant. Try playing around with loop delay time and see what works. Better yet, change your encoder to the new rotation sensor, which has 4096 ticks instead of 360. This means that even if you are losing accuracy due to low loop delay, you’ll only lose at most 0.088 degrees instead of 1 degrees, which is probably insignificant enough to cause any issues (0.088 deg on 2.75” tracking wheel is 0.002”)