IME - Technical description and software workarounds

[jpearman]

The VEX Integrated motor encoders have been available for over a year and we are still experiencing some frustrating technical issues. So that teams may understand why failures are happening, I’m going to present some information to explain how they work and what can cause software that is using them to become lost. I am not going to explain the details and differences of the ROBOTC and EasyC implementations, however, the details of the communications protocol were made public last year, and I have provided an implementation for the VEXpro controller which is open source, so none of the following is proprietary. This post will be in more than one part, part1 is covering the basics of the communication protocol.

The integrated motor encoder uses a very different connection method to the other VEX sensors that is known as Inter Integrated Circuit or I2C. Phillips invented this interface in the early 1980s as a low cost means of extending the IO capabilities of the simple microprocessors being used in consumer products at that time. The I2C bus uses synchronous serial communication with only two wires, a clock generated by the host (or master) device is used to control the transfer of bits on a data line either to or from a slave device. The two wires are typically called SCL for the clock and SDA for the data. In the four wire cables supplied by VEX, the clock is on the yellow wire and data on the white wire. The other two wires, red and black, supply power to the IME with the black wire also acting as the common ground reference for the system.

The I2C master always initiates a transaction, in our case this is the cortex, every slave device needs a unique address (think of the address of you house) so the master can talk to it. In many designs these addresses are fixed or set by configuration pins on the particular device, however, as I will explain later it works a little differently with the IMEs.

To allow bi-directional data transfer SCL and SDA are driven by open-drain drivers. The wires are pulled high (to the positive voltage) by using resistors, the master or slave device can only pull the wire low (to 0 volts). A typical I2C bus looks like this.

Data is transferred in 8 bit units called bytes, a data bit is placed on the SDA wire and the clock wire pulsed. A data bit should only change when the SCL wire is low, if SCL is high then the change on the SDA wire has a special meaning. If it changes from high to low this is called a START bit, if it changes from low to high this is called a STOP bit. Only the master device (the cortex) should create start or stop bits as they signify the start and end of a message.

So how does the master communicate with the slave? A typical sequence to send one byte to a slave would be as follows.

The master sends a START bit
The master sends the address of the slave it wishes to communicate with (normally 7 bits but it can be 10)
The master sends 1 bit indicating if data will be transferred to or from the slave.
The slave acknowledges it is ready by pulling SDA low for one clock cycle
The master sends the data byte 1 bit at a time starting with the msb (most significant bit)
The slave acknowledges it has received the data by pulling SDA low for one clock cycle.
The master sends a STOP bit.

IMEs differ from most I2C devices because the number of devices on the bus can be different for each robot. A technique called dynamic addressing is used, when IMEs are first powered on they all have the same address, the bus master (the cortex) learns how many devices are connected and assigns a new address to each one it finds before normal communication is started. Because each device has the same address when first powered on, there also needs to be a method of communicating with each IME so the new address can be assigned, this is done by disconnecting the SCL and SDA wires from the I2C bus for all but the IME nearest to the cortex. After a new address has been assigned that IME allows the next IME in the chain to be connected. Here is a high level diagram showing connections, note that the yellow (SCL) and white (SDA) wires are switched inside each IME.

Initially each IME has an address of 0x60 (“0x” signifies a hexadecimal number), after the IME chain is initialized they will typically have addresses starting at 0x20 and incrementing by 2 (0x20, 0x22, 0x24 etc.). Once initialized, a message can be sent to each IME in turn asking for the current number of encoder pulses it has counted. A typical message would send one data byte from the cortex to the IME indicating what information is required followed by the cortex reading a number of data bytes. Using an oscilloscope a single message can be captured and looks like this.

The yellow trace is the SCL wire, the blue trace is the SDA wire. This message was reading 6 bytes of data from the IME at address 0x20. A close up look at the beginning of this transaction is as follows.

I have added annotations to this capture as well as vertical lines showing where the SCL wire is changing from low to high, this is where bits on the SDA wire are valid. The transaction starts out with a START bit (indicated by the red S) followed by the address 0x20 and a write transaction bit. The slave responds with an acknowledge bit where it holds the SDA line low (indicated by the green A). The master sends one data byte, 0x40, without going into details this means it wants to read the encoder count. The master now sends another START bit (known as a repeated start, this avoids having to send STOP followed by START) followed by the IMEs address again, it then reads 6 data bytes from the IME. As the master receives each data byte it acknowledges all except the last one where a not-acknowledge is used to indicate that all data has been received. Finally the master sends a STOP bit.

With all this as background, part 2 will explain some of the things that can go wrong with this communication and possible improvements we can implement in user code.

Further reading
IME protocol
I2C tutorial
connecting the VEXpro to IMEs
Official I2C specification

Photo of the 269 motor IME circuit board
Photo of the 393 motor IME circuit board

[jpearman]

Part 2.

So with the understanding of the previous post as background, what are the types of things that can go wrong with the encoders.

1. The 4-wire cable is removed from the cortex

The software will no longer be able to communicate with any IMEs. EasyC will continue to return the last known encoder count. ROBOTC will also to return the last known encoder counts.

2. The 4-wire cable is removed and then replaced into the cortex.

All IMEs will have been reset to the power on default condition. EasyC will no longer be able to communicate with the IMEs, they have all had their addresses set back to the default 0x60, and will return the last known encoder position. ROBOTC will reinitialize IMEs and encoder counts will reset to 0.

3. The 4-wire cable is removed from an IME that is not nearest the cortex.

All IMEs that lose power will behave the same as in case 1 or 2. Both EasyC and ROBOTC will return the last known encoder count for those IMEs.

4. The I2C communication is disturbed in some way not related to a cable being unplugged. This may be crosstalk from adjacent motor wiring or something of that nature.

As explained in post 1, the I2C communication is complex and relies on individual data bits being placed correctly with respect to the clock signal. The program in the cortex needs to take appropriate action if it detects an error situation and may or may not be able to recover. In the most serious cases, ROBOTC will reinitialize the I2C bus, which will mean all encoder counts are reset to zero.

5. Static discharge causes an IME to reset.

The IME contains a small micro-controller from STMicroelectronics (STM8S103F3). This micro-controller is susceptible to processor reset caused by static, in fact it has circuitry specifically designed to detect program corruption caused by static and then generate a system reset. If this were to happen (and I have no evidence one way or the other) then the IME would revert to it’s default I2C address and lose communication with the cortex. It would again be up to the communications software in the cortex to decide how to detect this and what action to take.

The most important thing to understand from all of this is that if an IME reverts to its default power on state with its default I2C address then generally the whole I2C bus needs to go through an initialization sequence that will reset all counts back to 0. If your software is waiting for a certain encoder count to be reached then at best you will move that motor too far or at worst (if the count has frozen) you will never reach it.

Debugging the IME reset problems on an un-tethered moving robot is hard. The majority of my tests have been performed using ROBOTC V3.51 and an unreleased beta version which may be available publicly soon. With V3.51 the most common problem is that IME counts first freeze and then reset back to zero. This is probably due to static discharge but generally it has not been possible to correlate IME reset with an ESD event. Improvements in the next version of ROBOTC have shown that some of these events were probably due to case 4 above, the new version shows roughly an 10x improvement, that is, the interval between IME resets was lengthened by approximately 10x.

To handle the remaining residual resets I am proposing that a software flywheel is used where if loss of communication with the IME is detected then the current encoder count is saved which, along with an estimate of lost encoder counts, is then added to the actual encoder count when communication is resumed. For example, if communication were lost when the count was at 1000 with the encoder stationary, then 1000 would be stored as an offset and added to the encoder count (which would be reset back to 0) when communication resumed. Pseudo code may be as follows.

IMEGetCount
{
    return the current encoder value added to the current encoder offset
}

IMEMonitorTask
{
    while(1)
        {
        if( IME communication is good )
            {
            if( communication was good last time )
                {
                Do for each IME
                    Save the current encoder count
                    Save the current encoder velocity
                }
            else
                {
                // We have just resumed communication
                Do for each IME
                    Estimate lost encoder counts based on the
                    velocity over that last 1/10 second
                    and add to encoder offset
                }
            }
        else
            // IME communication is bad
            {
            if( communication was good last time )
                {
                Do for each IME
                    Add the current encoder count to the encoder offset
                    make the current encoder count zero
                }
            Add 10mS to communications lost timer
            }
    
        Wait 10mS
        }
}

[jpearman]

Part 3.

Here is some preliminary code, this will not work yet for any released version of ROBOTC but gives you an idea of what’s involved. The function IMEGetEncoder replaces the call to nMotorEncoder. A single task is started by a call to IMEInit().

/*-----------------------------------------------------------------------------*/
/*                                                                             */
/*  Pre-release IME wrapper library                                            */
/*                                                                             */
/*  James Pearman - March 6 2013                                               */
/*                                                                             */
/*  This version is not finished but gives an idea of the plan                 */
/*                                                                             */
/*-----------------------------------------------------------------------------*/

typedef struct _ime_data {
    unsigned char   installed;
    unsigned char   delta_ptr;
    short           deltas[10];
    long            value;
    long            offset;
  } ime_data;

static ime_data _imes kNumbOfRealMotors ];
static TSemaphore  _ImeSemaphore;

task IMEMonitorTask();

void
IMEInit()
{
    if( getTaskState( IMEMonitorTask ) == taskStateStopped )
        {
        StartTask( IMEMonitorTask );
        // Give new task time to start
        wait1Msec(5);
        }
}

int
IMEGetSemaphore()
{
    SemaphoreLock( _ImeSemaphore, 2);

    short s = getSemaphoreTaskOwner(_ImeSemaphore);

    if ( s == nCurrentTask )
        return(1);
    else
        return(0);
}

void
IMEReleaseSemaphore()
{
    short s = getSemaphoreTaskOwner(_ImeSemaphore);
    if ( s == nCurrentTask )
        SemaphoreUnlock(_ImeSemaphore);
}

long
IMEGetEncoder( tMotor index )
{
    return( _imes[index].value + _imes[index].offset );
}

short
IMESetEncoder( tMotor index, long count = 0 )
{
    if( IMEGetSemaphore() )
        {
        _imes[index].value  = 0;
        nMotorEncoder index ] = 0;
        _imes[index].offset = count;
        IMEReleaseSemaphore();
        return(1);
        }
    else
        return(0);
}

task
IMEMonitorTask()
{
    TI2cStatistics i2c_stats;

    int     i, j;
    short   delta;
    long    value;
    long    lost_comms_timer;
    int     resume_comms_timeout = 0;
    ime_data *p;

    // Initialize resource semaphore
    SemaphoreInitialize( _ImeSemaphore );

    IMEGetSemaphore();
    
    // Clear data
    for(i=0;i<kNumbOfRealMotors;i++)
        {
        _imes i ].installed = 0;
        _imes i ].delta_ptr = 0;
        _imes i ].value     = 0;
        _imes i ].offset    = 0;

        for(j=0;j<10;j++)
          _imes i ].deltas[j] = 0;
        }

    // Learn which motors have IMEs
    // cannot do this in a loop as getEncoderForMotor only accepts constants
    if( getEncoderForMotor( port1 ) >= 20 ) _imes port1 ].installed = 1;
    if( getEncoderForMotor( port2 ) >= 20 ) _imes port2 ].installed = 1;
    if( getEncoderForMotor( port3 ) >= 20 ) _imes port3 ].installed = 1;
    if( getEncoderForMotor( port4 ) >= 20 ) _imes port4 ].installed = 1;
    if( getEncoderForMotor( port5 ) >= 20 ) _imes port5 ].installed = 1;
    if( getEncoderForMotor( port6 ) >= 20 ) _imes port6 ].installed = 1;
    if( getEncoderForMotor( port7 ) >= 20 ) _imes port7 ].installed = 1;
    if( getEncoderForMotor( port8 ) >= 20 ) _imes port8 ].installed = 1;
    if( getEncoderForMotor( port9 ) >= 20 ) _imes port9 ].installed = 1;
    if( getEncoderForMotor( port10) >= 20 ) _imes port10].installed = 1;

    IMEReleaseSemaphore();
    
    while(1)
        {
        // 10mS delay, don't change this
        wait1Msec(10);

        SensorValue dgtl11] = 1;
        getI2CStatistics(&i2c_stats, sizeof(i2c_stats));

        // skip if cannot get the semaphore
        if( !IMEGetSemaphore() )
            continue;
        
        // Check I2C communication good flag
        if( !i2c_stats.bI2CNeverResponded )
            {
            if(resume_comms_timeout == 0)
                {
                if( lost_comms_timer != 0 )
                    {
                    // Only add if comms lost for less than 1 second
                    writeDebugStreamLine("Comms lost for %dmS", lost_comms_timer * 10);
                    if( lost_comms_timer < 100 )
                        {
                        // compensate for lost communications
                        for(i=0;i<kNumbOfRealMotors;i++)
                            {
                            p = &_imes*;

                            if(p->installed)
                                {
                                // calculate delta for last 100mS of known encoder use
                                delta = 0;
                                for(j=0;j<10;j++)
                                    delta += p->deltas[j];

                                // add to offset an estimate of how many counts we lost
                                p->offset += ((delta * lost_comms_timer) / 10);
                                //writeDebugStreamLine("motor %d had %d added to offset", i, ((delta * lost_comms_timer) / 10) );
                                nMotorEncoder* = 0;
                                }
                            }
                        }

                    // clear timer
                    lost_comms_timer = 0;
                    }

                // For each motor
                for(i=0;i<kNumbOfRealMotors;i++)
                    {
                    p = &_imes*;

                    // Does this motor have an IME ?
                    if( p->installed )
                        {
                        // Read encoder
                        value = nMotorEncoder*;

                        // Calculate delta from last time
                        delta = value - p->value;
                        p->value = value;

                        // store in the delta history array
                        p->deltas[p->delta_ptr] = delta;

                        // increment delta history index
                        if(++p->delta_ptr == 10)
                            p->delta_ptr = 0;
                        }
                    }
                }
            else
                {
                resume_comms_timeout--;
                lost_comms_timer++;
                }
            }
        else
            {
            if(resume_comms_timeout == 0)
                {
                // First time around store new offset
                for(i=0;i<kNumbOfRealMotors;i++)
                    {
                    p = &_imes*;

                    // If IME installed
                    if( p->installed )
                        {
                        // Calculate new offset
                        p->offset = p->value + p->offset;
                        p->value  = 0;
                        }
                    }
                }

            // Wait 100mS after comms is resumed.
            resume_comms_timeout = 10;

            // 10mS more of lost comms
            lost_comms_timer++;
            }

        IMEReleaseSemaphore();
        
        SensorValue dgtl11] = 0;
        }
}

---

[jpearman]

Part 4.

Here is the release version of the code tested with the release version of ROBOTC 3.60.

I used the following code to test the library.

#pragma config(I2C_Usage, I2C1, i2cSensors)
#pragma config(Sensor, I2C_1,  ,               sensorQuadEncoderOnI2CPort,    , AutoAssign)
#pragma config(Sensor, I2C_2,  ,               sensorQuadEncoderOnI2CPort,    , AutoAssign)
#pragma config(Motor,  port1,            ,             tmotorVex393, openLoop, encoder, encoderPort, I2C_1, 1000)
#pragma config(Motor,  port10,           ,             tmotorVex269, openLoop, encoder, encoderPort, I2C_2, 1000)
//*!!Code automatically generated by 'ROBOTC' configuration wizard               !!*//

#include <ImeLib.c>

/*-----------------------------------------------------------------------------*/
/*  Test code for the IME library                                              */
/*-----------------------------------------------------------------------------*/

task main()
{
    char  str[16];

    IMEInit();

    clearLCDLine(0);
    clearLCDLine(1);

    while(1)
        {
        // Display encoder value
        // use IMEGetEncoder instead of nMotorEncoder
        sprintf(str,"%8d", IMEGetEncoder( port1 ) );
        displayLCDString(0, 0, str);
        sprintf(str,"%8d", IMEGetEncoder( port10 ) );
        displayLCDString(1, 0, str);

        // Press Btn 8 Up to clear encoders
        if( vexRT Btn8U ] == 1 )
            {
            // Use IMESetEncoder instead of nMotorEncoder
            IMESetEncoder( port1,  0 );
            IMESetEncoder( port10, 0 );

            while(vexRT Btn8U ] == 1)
                wait1Msec(10);
            }
        wait1Msec(50);
        }
}

Library is attached, let me know if anyone tries it and it works, it’s very hard to test this in a “real world” situation so use at your own risk. It’s almost the same code as post #3, just added some comments and cleaned it up a little.

.
ImeLib.c (11.1 KB)

[dtengineering]

An awesome explanation of I2C, and some great explanations of potential failures and consequences. All you young 'uns should go back and read it again, not just to learn about I2C, but to learn about how to describe a problem and propose solutions.

Where’s the “Like” or “Rep” button on this site?

Thank you,

Jason (who is only a “young 'un” to a very few people on this forum, I expect)

[jpearman]

Thanks Jason

One area I had neglected to look into when I originally wrote this description was the circuit used for the switch in each IME. When the IMEs are first powered, they all have the same address as I explained in the post #1. So that only one IME answers this address each IME has a means of disabling the I2C bus to the next IME in the daisy chain. I had assumed this would be an I2C bus repeater, something like the NXP PCA9515, it turns out to be a dual channel SPST analog switch from Texas Instruments. I was surprised at this choice due to the 10Ω series resistance, not the end of the world but there are parts with much lower Ron. Probably a choice dictated by cost or some other factor I’m not aware of, I never actually got around to reverse engineering the IME schematic.

[FullMetalMentor]

An excellent explanation! Thanks so much for sharing all of this.

[Team80_Giraffes]

Has anyone seen issues with the IME’s failing to communicate back in a timely manner?

There are a few other variables in the struct referenced in the library jpearmen has a few posts above about numbers of errors from the I2C connections of various types.

Does anyone know if one is a sum of all of the error types or if they are all independent? Also the struct seems to be for all the IME’s in the chain. Is there something available per IME/I2C item?

(I think we are seeing some flaky I2C response given the distances are not updating on one of our wheels. The wheel is spinning so it should be responding better. Looking at this code as a possible solution. Any back story on why the library was created with the estimated counts for missed items in the first place?)

[jpearman]

Has anyone seen issues with the IME’s failing to communicate back in a timely manner?

There are a few other variables in the struct referenced in the library jpearmen has a few posts above about numbers of errors from the I2C connections of various types.

Does anyone know if one is a sum of all of the error types or if they are all independent? Also the struct seems to be for all the IME’s in the chain. Is there something available per IME/I2C item?

The nPegCounts array are on a per IME basis.

typedef struct
    {
    unsigned long       nNumbOfSuccessfullPolls;
    unsigned long       nCumulativefFailedPolls;
    unsigned long       nNumbOfFailedPolls;
    } TI2CPegCounts;

typedef struct
    {
    unsigned long   nTotalAddressResets;                      // Total number of address resets

    // snip - variables removed for clarity

    TI2CPegCounts   nPegCounts[kMaxI2cDevices];
    } TI2cStatistics;

(I think we are seeing some flaky I2C response given the distances are not updating on one of our wheels. The wheel is spinning so it should be responding better.

It’s more likely to be a mechanical issue with the IME, open it up and see if there is any damage to the encoder wheel (gear).

Looking at this code as a possible solution. Any back story on why the library was created with the estimated counts for missed items in the first place?)

It was created to flywheel over an IME reseting due to static. I had seen occasional occurrences of an IME mysteriously becoming zero when it should have been non zero, the “guess” was that the ESD detection circuit inside the IME was causing a reset and setting the counter back to zero.

[Team80_Giraffes]

It’s more likely to be a mechanical issue with the IME, open it up and see if there is any damage to the encoder wheel (gear).

Thanks. I am afraid that may be the case but was hoping for intermittent connectivity.

At worlds last year, an interior motor screw that holds the actual motor in place gave way and settled in between the IME and the flywheel digging a nice groove in the paint. Needless to say, that one had issues too after that happened. I never tried that IME agian thinking it scratched the sensor too.

[Team80_Giraffes]

Thanks again for the code! I added a line to print the total error count variable to it.

Update - no screws loose, no detected dropped/error packets from the IME global perspective. No loose wires either.

There may have been some motor grease covering the sensor or wheel though. We gave each one a good wiping and it seems a bit better. Not perfect, but better.

Also added a sonar as a double check for some configurations.

[OscarMNOVA12]

Is there some way to workaround an IME not initializing? Particularly the first one in the chain? I’ve checked the wires they’re fully inserted the correct way on both the Cortex and the IME, yet still nothing. Can we manually address them?

[jpearman]

The question of why the IME limit is set to 8 in ROBOTC came up today. I don’t have a definitive answer but here are some of the factors that contribute to that decision.

1. Is it an addressing issue?

No, addresses are 7 bit and there can be as many as 126 devices on an I2C bus, address 0 is reserved and the host often also has an address assigned. Addressing does not limit the number of IMEs to 8.

2. Is cable length responsible?

Not directly, but I don’t know what the limit for running a single IME on a long cable is as I have never tried one more than about 36 inches away from the cortex.

3. So what’s the issue?

The I2C bus is driven by an open collector circuit, pull-up resistors are needed to allow the signals to go into the “high” state, a transistor is turned on to pull them down to the “low” state. When designing an I2C bus the choice of the pull-up resistors is important and there is usually only one on each of the SDA (data) and SCL (clock) signals. Posts at the beginning of this thread explained how IMEs use an active switch to enable and disable signals to each downstream IME. Each IME has its own pull-up resistors, as IMEs are added to the chain the resistors will combine and the overall resistance will drop. The value of this resistor in an IME is 30kΩ, when two IMEs are used two 30KΩ resistors are now seen by the driver in parallel giving an equivalent resistance of 15KΩ, with 4 IMEs and we are down to 7.5KΩ, 8 IMEs and we are at 3.7KΩ. This varience in resistance means that the choice has been a compromise that allow the bus to work within defined limits.

The active switch in each IME also adds series resistance, far more so than the wire. The data sheet suggests about 10Ω for the “on” resistance of the switch.

In addition to the above we are also adding capacitance to the bus, this is the real issue, how much I don’t know but the end result is that what were fairly clean clock and data signals when using one IME have been degraded by the time we have added 8 IMEs.

Here is a scope capture showing the I2C clock when only one IME is being used.
[ATTACH]8733[/ATTACH]

The rising edge of the clock looks good, how fast the clock rises is determined by bus capacitance and resistance.

Here is a scope capture of the I2C clock (at the cortex output) when 6 IMEs are daisy chained together.

[ATTACH]8734[/ATTACH]

Notice how the rising edge of the clock is now much slower (The same effect is also happening to the data line but the clock is easier to capture).

The I2C spec says that in standard I2C mode (that is 100KHz clock which ROBOTC, EasyC and ConVEX all use) the SDA and SCL lines should have a maximum rise time of 1uS. This second scope trace shows that with 6 IMEs we are pushing the limit on this spec. The addition of two further IMEs will slow the transitions even further and I suspect was the limiting factor in how many IMEs can be added to the chain.

Now having explained the above, it may just have been that 8 seemed like a reasonable number that no one would exceed in normal use

[Team80_Giraffes]

Another awesome explanation from jpearman!

What happens when the signal gets too squashed for the time frame? Is it a full reset of the cortex or is it dropped packets?

It’s been a while since you whipped out the oscilloscope?

[FullMetalMentor]

Another awesome explanation from jpearman! …

Agreed.

…
It’s been a while since you whipped out the oscilloscope…

You mean… he doesn’t carry it around with him wherever he goes?

[jtarlo]

Hey I’m trying to read the vex integrated encoders off of a raspberry pi with i2c, so I am not using RobotC or anything. I was looking at the vex wiki for the encoders but am having trouble figuring out how to read the encoders with only the i2c addresses. Do you know how to do this? I could not find any information regarding reading these without the actual robotC software. I have already been able to initialized the encoders and they blink green about every 3 seconds.

[jpearman]

IME device spec is here.
https://vexforum.com/t/diy-vex-lcd-display/15080/1

There is some open source code that communicates with the IMEs here.

jpearman/convex

there was also an implementation for the (now discontinued) VEXPro controller in this thread.
https://vexforum.com/t/vexpro-arm9-i2c-support/22078/1