Field control - a technical analysis

There’s no real way to determine much definitive from video. Plus, I don’t have time to review them all in detail.

I will say in that first video though, you can see the LEDs on some of the controllers and they are definitely red at the moment of loss. Red is an error condition, obviously. It is not the field disabling things as that is yellow, not red. I also noticed that it seems to happen at the exact instant that a robot hits the wall fairly hard. I could imagine that happening if there was a static discharge when the robot contacted the alliance wall (again, I really don’t know if that’s the cause at all, but it could possibly explain it).

Of course, that puts EPs in a difficult spot because I’ve seen a lot of people complaining about the use of static spray too.

Dave, beside the James’ original analysis in the beginning of the thread, your post is the most complete and informative on this topic of (random) simultaneous disconnects. And it strikes the right tone too. Thank you!

I think, others would agree that the leading probable causes for the alliance disconnects are:

  1. Damaged competition port on one of the joysticks (intermittently shorting signals)
  2. Damaged cables connecting Match Controller, Driver Interface, and Joysticks (disconnecting or shorting)
  3. Unknown issues with connection between Windows and USB interface on Match Controller as @calvc01 describes in his post.
  4. And finally a chance that ESD is injected anywhere in the system causing Joystick circuitry to interpret erroneous control signals. This is a wild card as it may depend on the field conditions, air humidity, plastic parts used on one of the robots and/or field, and even quality of the main power source at the competition and power supply filters in the control computer.

I wish we heard from the VEX engineers who have all the internal knowledge of the control system hardware and software.

I’ve seen that before, though not in many years. When I’ve seen it, it’s seemed like a static issue - I’ve had it happen when a student picks up the field control cable before a match and gets a static shock from the cable. It’s pretty likely that the CP2103 chip on the field controller reset or otherwise entered some bad state at that point. It obviously disconnects from the USB (hence the “da-ding” Windows sound). My memory is not great but I thought you even had to unplug the field controller and plug it back in to recover. Maybe that’s only in some cases.

Like you say, that one’s an obvious replay because nothing works until you stop the match and fix it. I’m quite certain that’s not the cause of anything being discussed here, just because it is so disruptive that it’s much more obvious.

Alright, here is my few cents as to what I think what is going on, while discussing with the team 590, who has a field themselves.

After going through my days, both my PC and my laptop that has Windows 10 do have times where a USB device disconnects and reconnects, and yet it still maintains power to the USB device. Usually, the time when the USB device disconnects and reconnects are similar in time with the robots that disconnect in these competitions. Keep note, this is unofficial, but I think it has to do with the fact that the computers are running Windows 10, as well as the field control is not connected to the main USB port. After discussing with 590, they have said that they have had absolutely no problem with the field control running Windows 7, but they will be connecting a Windows 10 computer and run some tests to see if this is the actual case.

TL;DR - The field control problem may be with the fact that the fields are not being ran on the main USB port on a computer that runs Windows 10.

We have a full VEX Worlds field set up here at IFI HQ and will be trying to simulate the conditions that could cause issues. We have done this many times in the past and have been unable to replicate many of the failures described in the various threads. I am not saying they didn’t happen, just that we have been unable to replicate. We have an ESD gun here so we will use that to simulate high voltage ESD instances as well.

As far as VEXnet (which is different than all of the other field control items that have to do with wires, etc.), we have tens of thousands of hours of development time in the VEXnet system and spent millions of dollars on product development of this specific technology. There is no other commercial (I can’t speak to military) system out there that can do what VEXnet can do. It is one of the most reliable wireless systems available for our use case. As with all wireless technology around the 2.4 band there are ways to create so much traffic and noise that no wireless system works, but our testing has shown that VEXnet will survive almost all broad spectrum noise issues.

The cases of simultaneous alliance (one alliance) disables I have witnessed have all been related to a bent pin, or foreign metallic debris in the controller competition port. again, I am not saying there couldn’t be other causes, just not ones I have seen.

We take the customer experience very seriously and while we may not agree with every point of feedback from every single customer, we try to do what is in the best interest of the community at large. This year I think I will just record our Wednesday key volunteer reception speeches so you can hear it for yourselves.

I look forward to seeing everyone attending VEX Worlds. If you see me, please come up and say hello.

Thanks for the transparency Paul. I’m sure everyone in this thread would be interested to hear any findings with respect to ESD testing etc.

It’s very encouraging to know this is being looked into by the right people. I’m sure I’m not the only person who just added “Check competition port on joystick and plug on field control cable” to my pre-match checklist.

This is a separate issue that doesn’t cause disconnects. We had this issue with some laptops (Thinkpads) running Windows 10, where it would lose connection to the field control. In general this is because of Windows 10’s USB power-saving functions, but even when disabling those the issue would still persist. I should note here that no laptops running Windows 10 other than the Thinkpads would continue to have this issue after disabling the appropriate settings. The “fix” that we ended up using was to just use a powered USB hub for the field controllers.

When this issue happens, one alliance gets disabled, while the other remains in the current state indefinitely (usually driver control). It does not cause VEXNet disconnects, and does not just fix itself after a few seconds - for us it required re-plugging the field controller back into the computer.

I had a working theory that this may be a potential cause.
I also setup a full field control system earlier today and have run through a number of tests trying to duplicate single alliance disconnect, I have been unable to achieve that so far.

Yeah, I don’t think it’s anything related to the computer side. Like I said, any time that specific issue happened for us it never resulted in a VEXNet disconnect, only a disable (field control would of course reflect this).

I also took apart a joystick last night to investigate, and tried to cause VEXNet to drop by messing with the competition port, but couldn’t find anything. We also don’t have an ESD gun here to try and force any issues related to that, but maybe @Paul Copioli can investigate further since they do.

If ESD has anything to do with these loss-of-control events then it could be significant what type of fabric drivers were wearing. Some fabrics could be a source of the significant charge buildup.

Also, I would assume the test items (field controls, joysticks, cables) that @Paul Copioli and @jpearman are using for their tests are in a good shape and with clean connectors.

However, I bet, randomly selected joystick would accumulate all sorts of foreign material in its competition port over the years of use. Similarly, students coming to play after the launch and picking up their end of RJ45 connector could leave all sorts of greasy residue on the terminals.

One of my college professors would use yesterday’s pizza leftovers (no kidding) to “properly simulate” real world effect on some of the consumer electronics he was testing. If there was a pluggable module of any sort he would push it into slice of pizza first, then plug it into device and see how it would perform.

Because you never know what happens with input buffers when ESD event occurs, if the ground terminals have an extra layer of grease.

First, thank you for your response. It is really helpful to know you are listening to our feedback and are trying to determine the cause of the issues.

Here is a picture of my controller. There is no damage that I can see besides the absence of the tab that holds the cable in (a task that my partner controller mount incidentally fulfills).


Of course this doesn’t rule out my partner’s joystick being damaged. However I would think that shoring pins in the joystick would cause constant rebooting or other issues, and not the once in a match issues we are seeing.

In watching the videos from the US Open, it looks like they were primarily on the red alliance on one field. If that is the case, I am wondering if it may be a field wiring issue.

2 of the disconnects were on the red alliance on the right field in black division.
1 of the disconnects was on the red alliance on the middle field in black division
1 of the disconnects was on the red alliance on the middle field where the open matches were run (this is the grand finals match)

While three separate fields and two computers were involved, so it is unlikely that that is the problem, it is intriguing that all disconnects happened on the red alliance. However I don’t really see a way they could be connected unless red is somehow wired differently from blue (which AFAIK is not true). So at this point I’m just putting that down to coincidence.

@sazrocks thanks for the clarification. It well could be that if most of the alliance disconnects throughout the say were all red, that the cables running to the red alliances were all somewhat compromised. That is still coincidence as far as them being red, but it would explain why all are at the same 3 alliance stations rather than it happening at all the alliance stations.

Definitely possible.
I would like to point out though that these videos were pulled from videos of my matches (10 matches plus elims; so like 5% of the matches in black division) and my friend’s two videos of the finals, and no more. It is entirely possible that there were disconnects on the blue alliance but they just didn’t happen while I was filming.

I don’t know exactly what matches you were watching, but In the MS Division field 2 had way more issues than the other two. In addition, it seemed as if the red alliance in the MS Division had more “issues”, but that is just my observation, so I have no proof of that.

Thank you for the detailed response. GREAT info here. It is reassuring as I do 99% of what you recommend and we have few issues. The one thing that you mention that was new to me though was that a bad controller can cause yellow/not-green on the driver station?

We ran over 1300 official VRC qualification/elimination matches this year and never had an alliance go down either (except for the obvious person that kicked the extension plug out of the wall). I’m trying to train my kids to look at the driver’s station when they suspect and was a bit proud at the CREATE event last weekend when one of them did, pointed it out to the refs/techs, and got a replay of the match because of it.

By controller do you mean joystick? If so, yes, a shorted pin in the competition port on a joystick can disable both robots and change the driver station to yellow (though the yellow light will be solid rather than slowly blinking).

If you mean the field controller, then I don’t really know for sure. I’ve never had a field controller fail. If one did I think it would be really obvious, and likely not just causing a few seconds of control loss.

In a similar vein, could a shorted joystick caused an alliance driver station to enter a state other than disabled?

At VA states it appeared that due to crossed pins on one of my team’s joysticks it kept their alliance stuck in auton and unable to move. I was wondering if the short can cause a disabled state to appear whether it can force it to other states.

Of course, once they switched it out that night it wasn’t an issue afterwards.

Sorry, I did mean joystick. Good to know. It will likely be less of an issue with the V5 and many teams using new equipment next year, but there will certainly be teams still using the cortex - so that is good to keep an eye on. I know that my organization really pushed the life of all the hardware as we continued to wait patiently for the V5.