Field control - a technical analysis

This is a separate issue that doesn’t cause disconnects. We had this issue with some laptops (Thinkpads) running Windows 10, where it would lose connection to the field control. In general this is because of Windows 10’s USB power-saving functions, but even when disabling those the issue would still persist. I should note here that no laptops running Windows 10 other than the Thinkpads would continue to have this issue after disabling the appropriate settings. The “fix” that we ended up using was to just use a powered USB hub for the field controllers.

When this issue happens, one alliance gets disabled, while the other remains in the current state indefinitely (usually driver control). It does not cause VEXNet disconnects, and does not just fix itself after a few seconds - for us it required re-plugging the field controller back into the computer.

I had a working theory that this may be a potential cause.
I also setup a full field control system earlier today and have run through a number of tests trying to duplicate single alliance disconnect, I have been unable to achieve that so far.

Yeah, I don’t think it’s anything related to the computer side. Like I said, any time that specific issue happened for us it never resulted in a VEXNet disconnect, only a disable (field control would of course reflect this).

I also took apart a joystick last night to investigate, and tried to cause VEXNet to drop by messing with the competition port, but couldn’t find anything. We also don’t have an ESD gun here to try and force any issues related to that, but maybe @Paul Copioli can investigate further since they do.

If ESD has anything to do with these loss-of-control events then it could be significant what type of fabric drivers were wearing. Some fabrics could be a source of the significant charge buildup.

Also, I would assume the test items (field controls, joysticks, cables) that @Paul Copioli and @jpearman are using for their tests are in a good shape and with clean connectors.

However, I bet, randomly selected joystick would accumulate all sorts of foreign material in its competition port over the years of use. Similarly, students coming to play after the launch and picking up their end of RJ45 connector could leave all sorts of greasy residue on the terminals.

One of my college professors would use yesterday’s pizza leftovers (no kidding) to “properly simulate” real world effect on some of the consumer electronics he was testing. If there was a pluggable module of any sort he would push it into slice of pizza first, then plug it into device and see how it would perform.

Because you never know what happens with input buffers when ESD event occurs, if the ground terminals have an extra layer of grease.

First, thank you for your response. It is really helpful to know you are listening to our feedback and are trying to determine the cause of the issues.

Here is a picture of my controller. There is no damage that I can see besides the absence of the tab that holds the cable in (a task that my partner controller mount incidentally fulfills).
Of course this doesn’t rule out my partner’s joystick being damaged. However I would think that shoring pins in the joystick would cause constant rebooting or other issues, and not the once in a match issues we are seeing.

In watching the videos from the US Open, it looks like they were primarily on the red alliance on one field. If that is the case, I am wondering if it may be a field wiring issue.

2 of the disconnects were on the red alliance on the right field in black division.
1 of the disconnects was on the red alliance on the middle field in black division
1 of the disconnects was on the red alliance on the middle field where the open matches were run (this is the grand finals match)

While three separate fields and two computers were involved, so it is unlikely that that is the problem, it is intriguing that all disconnects happened on the red alliance. However I don’t really see a way they could be connected unless red is somehow wired differently from blue (which AFAIK is not true). So at this point I’m just putting that down to coincidence.

@sazrocks thanks for the clarification. It well could be that if most of the alliance disconnects throughout the say were all red, that the cables running to the red alliances were all somewhat compromised. That is still coincidence as far as them being red, but it would explain why all are at the same 3 alliance stations rather than it happening at all the alliance stations.

Definitely possible.
I would like to point out though that these videos were pulled from videos of my matches (10 matches plus elims; so like 5% of the matches in black division) and my friend’s two videos of the finals, and no more. It is entirely possible that there were disconnects on the blue alliance but they just didn’t happen while I was filming.

I don’t know exactly what matches you were watching, but In the MS Division field 2 had way more issues than the other two. In addition, it seemed as if the red alliance in the MS Division had more “issues”, but that is just my observation, so I have no proof of that.

Thank you for the detailed response. GREAT info here. It is reassuring as I do 99% of what you recommend and we have few issues. The one thing that you mention that was new to me though was that a bad controller can cause yellow/not-green on the driver station?

We ran over 1300 official VRC qualification/elimination matches this year and never had an alliance go down either (except for the obvious person that kicked the extension plug out of the wall). I’m trying to train my kids to look at the driver’s station when they suspect and was a bit proud at the CREATE event last weekend when one of them did, pointed it out to the refs/techs, and got a replay of the match because of it.

By controller do you mean joystick? If so, yes, a shorted pin in the competition port on a joystick can disable both robots and change the driver station to yellow (though the yellow light will be solid rather than slowly blinking).

If you mean the field controller, then I don’t really know for sure. I’ve never had a field controller fail. If one did I think it would be really obvious, and likely not just causing a few seconds of control loss.

In a similar vein, could a shorted joystick caused an alliance driver station to enter a state other than disabled?

At VA states it appeared that due to crossed pins on one of my team’s joysticks it kept their alliance stuck in auton and unable to move. I was wondering if the short can cause a disabled state to appear whether it can force it to other states.

Of course, once they switched it out that night it wasn’t an issue afterwards.

Sorry, I did mean joystick. Good to know. It will likely be less of an issue with the V5 and many teams using new equipment next year, but there will certainly be teams still using the cortex - so that is good to keep an eye on. I know that my organization really pushed the life of all the hardware as we continued to wait patiently for the V5.

Yes. I’ve seen the ‘stuck in autonomous’ happen because of a crossed pin in a joystick too in addition to ‘stuck in disabled’.

If you look at the pinout for the competition port (documented in the schematics at the top of the thread), you’ll see the first 4 pins are this:

  1. Ground
  2. Driver/Auto#
  3. Enable/Disable#
  4. Ground

If pins 1 and 2 are shorted, you will be stuck in auto mode. If pins 3 and 4 are shorted, you’ll be stuck in the disabled state. This happens because the field controller is providing a +5V signal on the driver or enable pin, but this signal is fed through a resistor on the field control board. When the pin is shorted, then the joystick Driver input (or Enable input) has a direct connection to ground (0 ohm) and also a connection to +5V (through a 100 ohm resistor). The path of lower resistance wins, so the joystick reads the signal as 0 volts, which indicates Auto or Disable depending on which pin is shorted. For some reason, the 3/4 short has appeared more often in my experience - I suspect this might be the result of people plugging in the wrong RJ connector (from the partner cable or programming cable) into the competition port. Since those connectors are narrower, the plastic on the outer edge might compress the pins in the competition port further until they reach the end of the little slot that holds them in place which then allows them to jump to the neighboring slot.

Shorted pins in the joystick is usually somewhat easy to diagnose because of a side effect of the driver interface design. As detailed before, each driver interface has a single connection that provides power to the LEDs on the board. When we implemented VEXnet support in Tournament Manager, we decided to turn that power on and off to create an effect of blinking on the driver interface LEDs similar to the Game LED on the joystick (I think the game LED used to blink yellow slowly for disabled on the joystick, now I believe it blinks rapidly instead). Since the joystick can’t short out the LED power, the on/off status of the LEDs will still match whatever state Tournament Manager is commanding for the field. Thus, if the match is supposed to be in driver mode and you see the auto light on, but it remains on steadily, that’s because TM is sending the driver state and not toggling the LEDs (because the joystick game LED remains solid on during driver mode). This same thing can happen if the disabled pin is shorted - the yellow disabled LED will be on, but during autonomous, TM will be toggling the LED power rapidly, causing the yellow disabled LED to flash in a way that it otherwise doesn’t (a rapidly flashing yellow disabled LED on the driver station interface is a dead giveaway of a short and the easiest one to spot).

Okay after reviewing the disconnect footage again, I noticed something interesting. In every video where the vexnet key is visible, its light is solid blue for some length of time during the disconnect. This appeared odd to me since that doesn’t happen during normal operation.

Will update this post with results of testing scenarios to generate a solid light.

-Ok, simulating a disconnect robot side (by power-cycling the cortex) did not yield the solid blue light.
-Simulating a disconnect joystick side (by power-cycling the joystick) did yield the solid blue light.
I will attempt to procure a solid blue light by using only the field control and will update this post with results.

Update two:
-Starting a joystick and cortex not connected to field control and then connecting them to field control (specifically when pin 6 is connected to signal the change to competition channels) did yield the solid blue light.
-However, disconnecting pin 6 or disconnecting from the field control all-together does not yield the solid blue light. This leads me to believe that this is not what is happening for the videos.
I also tried shorting different pins together, this however yielded no unexpected response from the joystick of any kind.

Therefore, based on the evidence above, I believe it is one of two causes:

  1. A field control-caused joystick crash/reboot, whether that be through static or another cause.
  2. Some other factor that would cause the vexnet key to immediately seek out a new channel.

#2 brings in the possibility of wireless interference, however considering how astronomically unlikely it is for a source of interference to simultaneously disconnect two vexnet links, I believe the first explanation is the most likely at this point.

I would also like to point out that the main processor in the joystick, the NXP LPC2458FET180, is the same processor that is used as the system processor in the Cortex. The cortex’s processor, as we all know, does not handle static well.

While I can’t specifically speak to the reliability of VEXnet keys, since I’ve only experienced one true drop-out, in an area with an extremely crowded & active 2.4 band, I can speak to the reliability of devices on the 2.4GHz band in a completely different field, where we experience extremely odd disconnects without having a real diagnosed cause.

Let’s take one of the most reliable Wireless DMX Rx/Tx systems on the market, the City Theatrical SHoW Baby 6 as an example. At this point I’ve worked with about 30 or 40 of these devices in a wide variety of configurations, and what I’ve found to be the case is that their performance in an empty theatre or arena is far superior to that of a filled venue (but by no means does a filled venue cripple this system), I would assume the same to be the case with VEXnet 2.0. These devices utilize frequency hopping, as I’d assume the VEXnet system would do, and for the most part, these units don’t have real connection dropouts, but every once in a blue moon, I experience drop-outs that are almost always solved by terminating the DMX run (which is achieved by running a 120 Ohm resistor across our data pins at the end of the chain), and in-turn reduces the overall system noise. It’s generally accepted that DMX termination isn’t necessary, but a luxury, but is also the first point of failure I look to when solving an issue with a lighting system.

To the point of no other system like VEXnet existing, I suggest you look into some of the wireless DMX systems, they’re definitely not for robotics, but are some of the most robust 2.4 GHz products I’ve ever encountered. The same company as I’ve linked above, City Theatrical, has released a product that can handle 8 concurrent, wireless DMX universe streams at the same time, which is an insane amount of data. (Multiverse)

I have no knowledge of the internals of the competition controllers and how they may terminate data, but is there a chance that there is some form of internal termination issue somewhere in the chain (Comp Controller => VEX Controller => VEXnet => Cortex)?

This thread was loads of help working on my new project. its not finalized yet any I would appreciate feedback from more experienced users. Especially on the emergency stop and display code…

A bit of info,

My project is making a field control system that is operated with buttons rather than the TM software. Check it out on git-hub at Jchisholm204/RasPi-Vex-Field-Controller

1 Like

Its now at Jchisholm204/ModFC

1 Like

@Jchisholm204 Interesting project. Some of us had kicked around ways to so a stand alone field in 2014-15. I ended up using an Adunio with 8 segment LED displays to do a count down. The interface was dirt simple. Press one button for Auto and it would count down 20 seconds. It would turn the team interface to Auto, count to zero and stop. Press the next button and it would do the same for Driver mode. Third button set the robots to disable and zeroed out the display.

You might want to reach out to the people that write the Tournament Manager Software (like @Dave_Flowerday)

Presently TM runs a remote display, you can plug the match controllers into the USB port and then the driver interface plugs into the match controller. I don’t know if there is a way for the Pi to do any local GPIO to TM.

Good luck with your project!