The micro-controller in the cortex has a theoretical maximum number of general purpose IO lines, the datasheet lists this as 80. These are separated into 5 groups of 16 labeled PORTA through PORTE. Many of these IOs are shared with other more dedicated functionality such as serial ports, this reduces the available IO to a much smaller subset.
The external interrupt controller on the cortex can use up to 16 external interrupt sources, any of the GPIO lines can be configured to be an interrupt source, however, the same bit from two different ports cannot be used at the same time. For example, you cannot use PORTA bit 0 and PORTB bit 0 at the same time as an interrupt source.
The digital IO on the cortex is mapped as follows.
Digital 1 PortE-bit 9
Digital 2 PortE-bit 11
Digital 3 PortC-bit 6
Digital 4 PortC-bit 7
Digital 5 PortE-bit 13
Digital 6 PortE-bit 14
Digital 7 PortE-bit 8
Digital 8 PortE-bit 10
Digital 9 PortE-bit 12
Digital 10 PortE-bit 7
Digital 11 PortD-bit 0
Digital 12 PortD-bit 1
Notice how Digital 4 and Digital 10 are both using bit 7, this is the root cause of the problem, digital 4 and digital 10 cannot both be used as a source of interrupt at the same time. As the quad encoders rely on interrupts you cannot have an encoder using digital 4 and digital 10.
There is more bad news, I’ve recently noticed that with ROBOTC (V3.51) if digital 10 is configured to be an output then an encoder on digital 3&4 may not work, something in the configuration of digital 10 as an output disables some configuration register for digital 4. This is a bug and I need to look at it some more, I don’t know if it’s a problems with other versions of ROBOTC.