Careful firmware updates

You have all seen that. Out of blue sky (preferably in the middle of a tournament), a working robot claims it needs an update.
And sometimes not even update helps and you no longer see a device. Lost forever.

So, I have figured out the second half of what’s going on. Read on for technical details, but TL;DR (which I have taught my team now) is:
If you see an unexpected upgrade request, STOP! Detach everything and check the devices one by one, never having multiple devices connected at the same time during such a the firmware update.

Details (some deeply technical):
We have lost a few devices this way, last time it was a motor. Few days later, we have seen the message again, but this time I was more careful and checked twice. The firmware upgrade utility claimed there are 2 color sensors to upgrade, but the robot only had one. The setup was: motor on 7, bumper on 8 and color sensor on 9.
I have unplugged the color sensor and everything else was fine. I have then flashed the color sensor on a dedicated brain and it was fine afterwards too.

That got me thinking and I came with a theory that the firmware updater, last time we lost the motor for good, might have flashed color sensor firmware into the motor, disabling it forever. With a motor not responding to any revival attempts, I had nothing to lose and cracked the motor open to verify my theory

  • . The MCU inside the motor is normally updated through the VexIQ cable with cooperation of the existing firmware (bootloader), but the MCU has other means of programming, enter JTAG. I have quickly located the JTAG programming header on the motor PCB and fetched the MCU content. One of the rare texts inside the binary was "**228-3012**". There you go! I have extracted proper motor firmware from the Firmware.db (located in appdata folder of the VexIQ firmware updater), flashed it onto the device and, Voila! motor works again! I am trying to figure out if there is a way of fixing such a device in a non-intrusive way (I have another such motor and a gyro in this state), but I don't give it much chance...

    Edit: I almost forgot: Big thanks to James Pearman for his reference sensor work, it helped me tremendously understanding the inner workings of the sensors.

  • I have it since confirmed from Vex engineering: "Yes, in rare circumstances, a misbehaving device on each bank of three ports can cause the other devices on that bank to become "confused" as well. Removing the other devices on the bank can help isolate the problem, because it is possible to brick certain devices if the wrong firmware is downloaded."

  • Nice reverse engineering! Nice writeup on what is going on with the possibility of bricking a device. I’ve got 6 kits here and I’ve been careful to plug just motors in, just color sensors, etc. when I flash them. I’ll put in my parent instructions about new code at events to unplug the other devices.

    I have peeked into the bootloader binary and it looked like it would stay in and answer update commands in case the I2C_EN is pulled low on powerup.
    And yeah, it did!
    I have revived one such a bricked motor without opening it. One that was invisible to the brain:
    I have spliced the VexIQ cable between the motor and brain, shorted yellow and red, then powered up the brain.
    The familiar “update needed” screen popped up and I was able to flash fresh motor firmware in through the Vex Firmware Update Utility.
    The only difference was that after the update, the utility still claimed “update needed” (because it was kept in the bootloader), but after replacing the cable, the motor was just fine.

    So, at least the motors could apparently (or at least sometimes) be rescued this way (and it might be also possible to add this as a brain feature, say, always turn on with port1 in this bootstrap mode, check for attached device and if found OK, just reboot the device through the bootloader protocol).

    Sadly, this trick didn’t work (so far) on a gyro bricked in a similar fashion. I have yet to investigate that one deeper…