IME Analog Switch failing to conduct?

I briefly fell into this problem a couple of months ago . At the time I though we had fixed it, but we started having problems with IME’s failing to initialize properly the week after. I finally received some time to set down and start to debug and I have somewhat tracked it down the the Analog switch connecting the SDA and SDK lines. As when I look at it on a scope there is no active on either line when this problem exists This issue is common enough that we encountered it every competition and once it starts it seems that the only fix to let it sit there for a while. The easiest way to trigger this event is to switch the brain off as soon as it starts to turn on. It will probably take a couple of attempts, but it should happen eventually. I know that this is an edge cause, but we’ve found that it is relatively easy to concurrence during competition as there are sometime intermediate power interupts from crashing around and other issues. Whats really weird about all of this is that when looking at the LED codes on the IME’s the second IME seems to believe that it has a valid address and double green blinks even in this error state. I am way over my head on this one so hopefully with the information I collected someone will be able to take a better guess at it than me. If it is important we are running the latest version of Convex.
Working IME's.png
IME's Not Working.png

So are you running the latest version from Nov 2016 where I made some improvement to this issue.

pretty weird that the leds think the IMEs are working ok.

Yes we are after the last fix that we talked about a couple of months ago the incidence rate seemed to go down, but it’s still there. Also after the problem arises it last for multiple power cycles with the only thing that solves it being leaving it off for a couple of minutes. Does that point to it being an issue with the IME’s themselves?

It’s hard to say, I would have to do some tests to see if it’s a hardware issue or firmware. RobotC does initialize a little differently than ConVEX but it’s not something I’ve ever encountered before. My own demo robots both have IMEs, one has 5 the other 3, and I have not seen this problem. Will you be in Louisville?

Just as an FYI to this. I have had to review a number of customer designed I2C issues in the past. I don’t have the schematic for the one you are discussing but one common issue with I2C is lockup. A functional example would be a system with a processor that is an I2C master and an “dumb” I2C slave chip, like a memory device. If the master is talking to the slave it controls the clock (aside from clock stretching), so the issue arises when the master requests a read from the slave and the slave starts to deliver the data from the read using the master clocks. If there is a power glitch in this sequence that cause the master to reset, but not the slave, when the slave is outputting a “low” value then when the master reboots, the slave is still driving the I2C data line low because it has been waiting for clocks to complete the transaction. When the master comes out of boot and tries to become the master, it sees that the data line is held low (it should always idle high) so it assumes someone else is the master and faults (depending upon how your software is setup). The way to fix this is to either have a power switch to all slaves that brings them during master re-boot, or you can turn the I2C clock line into a GPIO and force 9 clocks on the bus (or drive it until the data line idles high), our you can use an SMBus compliant chip that will time out under these conditions for you (since everyone had so much trouble with this).

yea, RobotC does this.

@TriDragon If it was a bus lock up wouldn’t the entire system go down not just one or two IME’s at the end?

@jpearman Just to throw a little more light on the behavior the fastest way to induce this behavior is to boot everything up then unplug an IME in the chain and replug it in (I know this will cause problems, but when we have issues the problem arises itself.) Also when this becomes an issue by itself the IME will work correctly for a few seconds or until the remote establishes connection with the cortex whichever is first.

Yes, probably the whole bus would be locked up, hard to say.

Perhaps you should send me your latest code so I can at least test with that and see if it’s something specific to your programming.

Probably, but I’m not sure. I’d have to see the schematics for the whole system to see.

Just as an aside, I have had a lot of problems with IQ I2C this year.

Yea I stripped the code all the way down to the example code . The only thing there is that the motor array now has the IME code in it. After some more testing it definitely happens when the controller connects to the cortex. I let it run for ten minutes and it was all fine, but as soon the controller connect it will sometimes drop an IME.

Ok, let me try and duplicate later this week.

Were you able to duplicate it?

Sorry, I’ve had no time to look at this.