From what I have seen, this is pretty common behavior for vex potentiometers. Even right out of the box, some just behave like this. Like others have said, you could try looking for a more reliable potentiometer, but you would probably have to test several of them.
I ran into a similar issue with the potentiometer on my tray angler. In this thread, I outlined the internal differences between an internal encoder and a potentiometer. I came to the conclusion that each sensor provides its own unique advantages and disadvantages. I came up with a solution that took advantage of each sensor’s input for more reliability.
Another conclusion of my experiment was the non-linearity of the potentiometer. Meaning if you turn a potentiometer θ degrees between
a and b and then φ degrees between c and d. θ and φ might be the same, but the difference between a->b and c->d would be very different. If you did the same experiment with the internal encoder the difference between a->b and c->d would be the same since encoder counts are designed to be equally spaced. Furthermore, the non-linearity is not uniform between each potentiometer, so that could explain why you see such large disruptions.
Maybe you could try using the delta in the internal encoder’s position to estimate the truth of the potentiometer value? If the internal encoder barely moves, but the potentiometer reads a huge difference then you know one of them is giving false values. Using the potentiometer to home the internal encoders is a good idea (it is what I used for my tray too), but could you try an alternative approach depending on the truth value of the sensors.