VEXnet Failure

So this past Saturday was the second most frustrating day of BEST I have experienced in the past ten years. Every round had robots not running because the VEX joystick lost comm with the Cortex controller, except for one preliminary round. We had comm in our first semifinal round and half of our second. We only had comm for one minute in the three rounds of the finals. We only had six out of fourteen rounds with radio comm for the entire round. To my knowledge, every team experienced similar issues, though we had our worst problems after the preliminary rounds.

Another team went through all three semifinal rounds without ever having radio comm. Yet another missed a round of the finals because of the lack of radio comm. Some rounds only had one of four robots running, all the result of the radio systems not linking.

My robotics teams always turn out good products and this tear was no exception. Several of the students and I read all we could on the VEXnet comm issues and tried averything. We even asked if we could use a USB extension to get our VEXnet key away from the Cortex but were told the cable was not in the kit of parts so we could not use it.

After what we went through Saturday, I just cannot imagine spending six weeks next year working with the kids to put a robot together to go through a similar experience. I teach a class in electronics and robotics. I used to fly radio control airplanes. If I am frustrated with the situation, imagine how teacher/mentors with less background feel.

VEX and BEST need to find a solution. The communication problems are well documented on the VEX forums.

Did you check for wifi interference in your area ?

We ran a VEX competition in Galveston this weekend with 16 teams. This event had the fewest problems I have ever seen in my 4 years of experience with VEX products.

I managed the field and we never lost one robot to communication problems throughout the whole event.

Could I ask a few questions.

  1. Was each team using it’s own VEXnet key for the joystick or was a USB cable used with a remote key?

  2. Do the BEST teams use the 9V backup battery connected to the cortex?

  3. Did you have WiFi internet access available at the competition?

We have experienced problems but have tracked our issues down to one or two USB WiFi adapters that behave differently to others.

I was also at the event this past weekend, and saw the frustration of the teams as they sat and watched their robots do nothing during the matches.

In an attempt to solve the communication problems they were having, the team in question changed out their VEX Cortex just before the finals, had it working in the pits, but when the matches started, they got no communication between the controller and the cortex again.

The field and pit set up was very similar to the Galveston VEX contest. Everything was in one gym, with the field being in the middle.

I think one of the problems was that the school did not turn off it’s wi-fi. I just don’t know if that would have caused the communication errors they were having. An IFI tech would have to answer that.

Since this is not like VEX, where you can register at another event and try again, this is a one shot attempt to try and advance to the Regional Championships. I understand the frustration of putting in those six weeks of work to build a working robot, and then to watch it fail on the field from some unknown communication error.

If it was happening to just one team, then I would think that the problem would be with something that they had built incorrectly, but since it was happening across the board to a lot of teams, then that leads me to think it was not something they had done.

Seems odd, I’m talking about the small PP3 9V battery, the battery holder for this is part of the VEXnet system bundle. We had a long discussion about the necessity of it here https://vexforum.com/t/9v-backup-battery/19645/1&highlight=backup+battery

Anyway, it sounds more like an environmental issue than anything else. There were some Wifi issues at worlds this year on the first afternoon but it appeared to improve after everyone turned off Wifi on phones, laptops etc. If the school WiFi was on this may have been a problem.

This was North Houston BEST?
Looks like the winners were able to score ~100 points a round.

From the BEST Kit perspective:

  • A USB extension cable for the Cortex USB vexnet key is an allowed optional component.

  • A 9V battery is provided; line item 4 under electrical in the C kit list for 2011

  • The 9V backup battery holder is line item 4 in the R kit list for 2011.

I have seen a surprisingly large number of defective vexnet wifi keys (which I have yet to get around to returning) last year. Some BEST teams spend relatively little time driving and exercising their wifi system, which helps to flush out the bad keys.

Its still a frustrating experience for the teams, I’m sure.

It was North Houston BEST.

The winning team was the one that had connection through all six rounds of the semis and finals. They only had connection for one round of the prelims.

We scored 653 points in the five rounds of prelims we had connection. We were the only team that scored all three types of insects and earn the 100 point bonus. We had the single highest score for a round at 163 points, but we were holding back during the prelims so we didn’t alarm the other teams. Sometimes they gang up on a high scoring robot to keep them from advancing to the finals. It has been attempted previously at our hub and happened to us at Texas BEST on year. We averaged over 200 points per round during practice and had a high single round practice score of 330.

I understand the USB extension was an allowed optional component. We asked to use it during the hub competition and were not allowed to. We tried anyway and it did not help. Nor did the 9 volt battery. Nor did everything else we try, including switching out the cortex, the joystick, the vex net keys, the robot battery, and the joystick batteries. Upgrading the firmware and synching the joystick with the Cortex didn’t help either. It was not a problem our team could solve. It was a problem that affected all teams, some worse than others. VEX and BEST need to find a solutionor have a contingency plan beyond saying “sorry, it won’t happen next year.”

Which communication problems are you referencing? I’d be curious to see the documentation you’re referring to. We’re constantly improving the integrity of the VEXnet link and working to enhance it’s functionality even in bad environments.

One thing we can’t fix…
Was there a wifi hunter-killer setup at the venue? There really isn’t a way around those since they target unauthorized wifi systems and bombard them to block communication.

If one of those systems is present the best solution is to either get the venue IT department to disable it, or alternately use the VEX MAC Address Utility to document the MAC addresses of the VEXnet keys in use, then get these MAC addresses white flagged.

-John

Hi John,

There are several posts on this forum detailing problems people have had with the VEXnet communication link.

https://vexforum.com/t/worlds-vexnet-dropouts/18849/1&highlight=vexnet+problems

https://vexforum.com/t/vexnet-keys/18427/1&highlight=vexnet+problems

https://vexforum.com/t/issuse-s-with-cortex-s/17939/1&highlight=vexnet+problems

[https://vexforum.com/t/answered-cortex-vexnet-connection-problems/18040/1&highlight=vexnet+problems

There are more.

A search on the BEST Forum will also turn up posts related to the issue.

I asked our campus technology person and he assured me our school district does not use wifi hunter-killer systems. The competition was held in a gym at one of our junior high school campuses. I do not know of any wifi issues at demo day which was held in the cafeteria of a local community college.

I wish you could have been at the site when the problems were happening. It was a fiasco. The problems were so bad I wasn’t even sure they would be able to have the competition. As a ten year veteran of BEST, I appreciate the increased capabilities afforded by the VEX system, but I can’t go through another six week season and have the same thing happen to my kids.

All I have gotten from the Texas BEST Regional Director and the BEST Executive Director is a “sorry” and we will pass the information on to the Kit and Game committees and to the BEST, Inc. Board. The Texas BEST director acknowledged that he has been hearing of problems and that they will be addressed and fixed by next year.

I can’t start BEST next fall without knowing the problem is actually fixed. As the mentor for a FIRST team, you know how much goes into preparing a robot in six weeks, even a BEST robot.

We have the same connection issue in our robotics lab on campus. The issue disappears if we take the robot outside or to the cafeteria. I would be happy to try anything you can suggest to find a solution. I will be happy to document exactly what I do so VEX can figure out what the problem is and how to prevent it from happening in the future. We have our own Cortex, joystick, etc. If we find a solution, I can drive out to try the system at this year’s competition venue where the problem was so severe.

My goal is to find a solution so this doesn’t happen to other teams and we can continue competing in BEST Robotics. I like the different experience my team gets from BEST after a long season of FIRST.](https://vexforum.com/t/answered-cortex-vexnet-connection-problems/18040/1&highlight=vexnet+problems)

Do you mind shooting me an email? I’d like to chat more about this.

-John
jvn@vexrobotics.com

I cannot offer a solution to your problems, however, I would like to comment on your VEXnet connection issues if others are reading this thread. So far this year the teams I mentor have attended two (VEX not BEST) competitions. We have not experienced any VEXnet dropouts or heard any complaints from other teams at the events we attended. This last weekend we were at the Viewpoint competition, 31 teams played 56 qualifiers followed by playoff matches. The setting was similar to that which you describe, a school gym, and to my surprise guest WiFi was available and being used by many people. I had expected interference but there were no problems that I observed.

I have often been asked to provide remote control WiFI networking solutions for the industry I work in. So far we have avoided providing this technology as we know that the environment it will be used in (movie and television production) is prone to all sorts of RF interference out of our control. The issues you are describing that involve multiple teams having problems sound like it’s an environmental issue. This may not just be other 802.11 networking but could be one of the other technologies sharing the 2.4GHz band.

I understand your disappointment and frustration, but think that the underlying technology used for VEXnet is as reliable as can be reasonably expected for a consumer level product.

I hope JVN can work it out with you, by putting him in direct contact with the campus IT support people.
My guess is that there was an ad-hoc network blocker, and that there is a communication disconnect with the campus IT person.

Dallas BEST has a new (to BEST using Vexnet) venue, the UTD gym, with game day on 10/29/2011. I think there have been Vexnet competitions there before, but I’ll make sure to check and see if we can get an IT person on speed-dial to support us on Friday night/Saturday.

I think we have to do better than watch the teams struggle with connection issues. Running the competition anyway was not the best solution. It may be that VEXnet is normally reliable but teams were unable to make connections between their joystick and Cortex. All the teams, lots of times.

This makes an interesting read.

http://www.cisco.com/en/US/prod/collateral/wireless/ps9391/ps9393/ps9394/prod_white_paper0900aecd807395a9.pdf

Stay away from microwave popcorn at events.

I’m not trying to minimize your problems but my point is that VEX uses commercially available USB WiFi adapters, their Ad Hoc networks are susceptible to the same interference as all other WiFi devices. They do not have control over the environment in which it is used.

I have been working with VEX Robotics, Inc. to resolve the issues with connectivity between the Cortex and joystick using the VEXnet system. After several people maintained that the problem sounded like the competition venue had a system that actively prevented “rogue” wifi systems, I checked with the coordinator of technology in our school district. He confirmed that they do use wifi controllers that prevent ad-hoc networks from connecting. I had previously checked with the local campus technology person and he told me the school district did not have such system.

I continued testing at home to make sure being away from the school would solve the problem. It didn’t appear to until I tried several VEXnet usb keys and found one of them was faulty. I am now working with our BEST hub to test all their VEXnet keys to eliminate them as a future problem.

The lesson I have learned is that wifi issues are not easy to resolve. Anyone hosting a competition that relies on wireless connectivity should make a thorough check of the site to eliminate that as a source of problems. You definitely don’t want to experience what the BEST teams did at our hub. If you would like more information about wireless interference, the link posted by jpearman is an excellent primer.

I had previously made the Executive Director of BEST aware of the hub problems. My followup with him will include the recommendation of throughly checking competition sites for wireless issues beforehand. Waiting to discover them the day of the competition is too late.

Many thanks to VEX Robotics, specifically John V-Neun and Ricky Torrance, for taking such a positive approach to finding the cause of the problem. BEST is lucky to work with such a proactive partner.

This message is also posted on the BEST Forum.