Field control - a technical analysis

I perfectly understand that, I’ve seen all of those happen with junior middle school teams. When I wrote that post I was thinking about high school teams that make it to Division Finals or RR. Those teams will protect their hardware even if they have to fall and get hurt themselves.

I’ve seen full alliance disconnects many times at local competitions. It usually happens on one side of one field a couple of times before everyone realizes it is the “unlucky” one and asks officials to do something about it. Wiggling or replacing wires fixes this most of the times.

I understand that at Worlds field control components are better handled and tested by professionals, but the videos of full alliance disconnects from Worlds and US Open finals prove that system failures are more frequent than anyone wants.

4 Likes

So the 8888 teams attended 5 competitions this year. I can’t remember the last time I saw a disconnect.

3 Likes

I believe there actually was a full alliance disconnect last year at the World Championship during the Round Robins, and yet nobody did anything about it. If anyone can clarify me wrong or if they have a video of the disconnect, it would be appreciated.

1 Like

The best advice I can give is try and collect information so the field techs on hand can assist you. What did your controller leds show, what did the driver interface show, could you see the led on the VEXnet key on your robot. But I understand that’s really hard during a match, it’s not always clear what is happening until it’s too late.

4 Likes

I saw my team disconnect at least twice in eliminations this season. One was brief 5-10 sec pause at the beginning of the driver control for both alliance robots. The other time both robots disconnected with about ~40 sec before the end. In the latter case it wasn’t first time that field had issues and the match was replayed on another field.

There might have be more disconnects during qualifications. I haven’t seen all games. I am usually busy doing pit judging during qualification rounds.

James, did you by any chance volunteer for any of those competitions and helped to set up the fields? :slight_smile:

A lot of hardware have “hidden” failure modes and what some consider “dumb luck” due to the lack of understanding of those modes is “proper setup procedures” for others, even though some of the steps make little sense for anyone else.

I wouldn’t be surprised if fields that you helped to setup worked perfectly since you know how fragile those RJ45 connectors are and would never pull or twist them in a risky way.

3 Likes

Actually no. I also don’t watch every match so there may have been issues I’m not aware of, just saying that since the switch to VEXnet 2.0, disconnects have become a bit of a non-issue for my teams. Perhaps it’s the warm weather out here :slight_smile:

4 Likes

James, since you mentioned VEXNet 2.0, do you know why IFI had recently submitted FCCID paperwork for VEXNet? I though it is being disconnected and there wouldn’t be any reason to update existing hardware.

Old: https://fccid.io/UKU-RAD03
New: https://fccid.io/UKU-RAD03B

Here is a video of me simultaneously disconnecting.

2 Likes

I don’t, and if I did it wouldn’t be something I could comment on.

2 Likes

I just looked through most of both filings, as well as the single Industry Canada filing (for the old version, RAD03).

The only difference I can find between RAD03 and RAD03B is that RAD03B was tested with a USB power bank, while RAD03 was tested with a Robot Brain, 7.2V/3000mAh Power Pack, and Remote Controller (presumably a Cortex, 7.2V 3000mAh NiMH battery, and VEXnet Joystick, respectively).

Given the lack of a new IC filing and continued use of the same 2013-dated “user manual” which calls for “suitable hosts,” my best guess is that IFI is merely responding to some regulatory or other change that warranted new testing. I found no evidence of any actual meaningful change(s), even after comparing photos of the internal circuitry.

TL;DR I think they just want to continue being able to sell their current stock of VEXnet 2.0 keys in the US without getting in trouble.

2 Likes

I watched that match closely a few times and noticed something interesting. It’s actually 3 of the 4 robots that all stop moving at the same instant, and begin moving again at the exact same time. The 4th robot kept moving the whole time. That tells us some interesting things. As @jpearman detailed clearly in this thread, there is no possible way the field can disable 3 of the 4 robots. That specific behavior rules out a lot of possible causes, including anything related to the field control system (it also means it’s not an “alliance disconnect”).

My first guess would be some sort of wireless interference. The 3 robots that stopped moving may have been on channels close to each other in frequency and it’s possible some other transmission on a nearby frequency interfered with all of them. I also notice that there’s a drive team member in a yellow shirt holding a phone, and while that person is obscured at the moment the robots stop moving, as soon as the camera shot changes he seems to be typing on the phone. It’s possible that the phone made a transmission on either WiFi or Bluetooth that disrupted the robots, especially if that person just sent a message or something like that. The red robot driver that was interrupted was also closer to that phone than the other red robot that kept moving. I also see a GoPro on the field ladder, those also have WiFi and could be the same thing. Note this is all speculation regarding a possible cause - there’s not enough information in the video to determine anything concretely.

If you’re looking for ways to help reduce the likelihood of an issue, my first piece of advice would be: don’t use phones in the alliance station. Ask your partners to do the same. Maybe even leave them with another team member while you drive. Would it have prevented this particular issue? Who knows. But it couldn’t really hurt.

1 Like

@Dave Flowerday You just told us a half a dozen ways how vex net can fail, this does not make me feel any better about the Bo1 situation.

If a product can just stop functioning because it is proximity to go pro’s, and cell phones, it’s not a reliable piece of tech. Because just about every human being at a competition is packing a cell phone.

Really good points Dave, I always tell our students they have to be in airplane mode at the fields.

1 Like

Please don’t try to sensationalize what I said. It’s not helpful. VEXnet is a wireless connection and I pointed out that it is susceptible to interference. This shouldn’t be news to anyone, and I absolutely did not tell you “a half dozen ways it can fail”.

ALL wireless devices have to deal with interference. It’s reality. The only alternative is a wired connection from your joystick to your robot. Robot competitions in the 90s used to work that way. Do you want to go back to that? I sure don’t.

VEX (note: I don’t work for VEX) puts in an amazing amount of engineering to make VEXnet as reliable as is possible in the face of reality. But, they use the 2.4GHz frequency band because they basically have no other choice. It’s one of the very few frequency bands that is available for worldwide use by unlicensed devices. Unfortunately that’s the same reason lots of other devices use it too, including Bluetooth, WiFi, microwave ovens, and tons of other stuff. VEX does lots of things to overcome these issues (most of which you and I don’t know about), and I know they work very closely with the manufacturers of the wireless chipsets used in VEXnet keys to get the best possible performance out of them. I know they take steps specifically to deal with and avoid WiFi interference. But, there’s a reason there’s 1000 signs that say “No WiFi” all over Worlds and why VEX brings in a bunch of equipment to detect and identify people using WiFi. And yet, when I walk around Worlds I constantly see people ignoring this rule (until the VEX staff with roaming WiFi detectors shuts them down, that is).

The link between VEX joysticks and the robots has one major advantage: they’re very close, only a few feet apart. Wireless transmissions (including interference) follows the inverse-square law, which basically means that the signal strength drops by 4x each time you double the distance between the wireless source and receiver. This means that someone sitting in the stands 40-50 feet away using WiFi is unlikely to have a strong enough signal at your robot’s position to interfere with it, even if they happen to be on the same channel. However, if a phone in the alliance station starts transmitting WiFi or Bluetooth standing 1-2 feet away from the joystick, that’s obviously way closer and thus could potentially be a strong enough signal to interfere. Hence my recommendation: don’t take devices which may contain WiFi or Bluetooth into the driver box. Even if you’re not actively using them, the devices may use their radios anyway even in your pocket. Just leave them with someone else; you can survive a 2 minute match without your phone.

1 Like

While you may not have have provided 6 ways it can fail you did list out several things I personally didn’t know would effect a vex net connection. Are you saying simple text messages and email can disconnect a robot from the controller?

I know @Raisors_Blade is being slightly dramatic, but going from only hot spots interfere, to maybe a sending a message will disconnect 3 teams is a big step and not a good argument for vex net’s reliability. It seems to me that Vex’s DSS is fairly weak. I have flown my drone (DJI MAvic pro) near the Kid’s robots and they go dead in the water, but never ever the other way around. I’m not saying my drone is perfect (it uses 5.8Ghz and two antennas (edit: maybe 2 2.4 Ghz channels, it’s not very clear to me based on my short research)) but their system of radio control is certainly more reliable and works from miles out.

We’ve even had up to four flying within a few feet of each other and they all maintained connection (both RC and HD video stream). Granted I know a company will invest in RC differently when its controlling a $800 aircraft, but I’d think Vex could get it sorted out for four robots at once less than 2 meters away.

1 Like

No, I did not say those things “would affect” a VEXnet connection. I said it’s possible. I didn’t say “definitely”, or “most likely”. I said “possible” for a reason. The post I was replying to was asking for things they could do to help avoid issues. I gave a possible explanation, and an easy thing for a team to do to avoid that possibility. It may not be a very likely possibility, but it’s an easy thing to avoid, even if there’s only a 1% chance of it being an issue (note, I don’t know what the chances are at all, but my semi-educated guess is that it’s not 0%).

Your anecdote about your drone is a great example of selection bias. Your statement implies that because you personally haven’t had issues with your drone’s wireless link that therefore that proves that the drone’s wireless link is superior to VEXnet. That could be true, but your lone experience does not prove that. The problem with that logic is there are certainly teams who have not had issues with VEXnet either. Likewise, a quick Google search shows that there are definitely people complaining about Mavic Pro dropouts. Doing a meaningful comparison of the two would be an excellent task for a statistics class.

It’s also probable that the drone is working with much different constraints than VEXnet. The VEXnet wireless hardware has to fit in a small USB key. Because of the size, and probably for other reasons, the wireless output power isn’t going to be very high. Certainly less than your drone, as your drone needs to be controllable from much greater distances than a VEX robot does. VEXnet doesn’t have external antennas like most drone controllers do, etc. Also, you reference running 4 drones at once and comparing that to a VEX event, but you’re forgetting that a VEX event has dozens of robots operating at the same time (in the pits, etc). At Worlds there could easily be 40-50 robots active on a competition channel at the same time, and dozens or hundreds active on other channels in their pits or on practice fields.

There were well over 1,500 VRC matches played at Worlds last year, or 180,000 seconds of match time. Finding a few instances where there’s 5-10 seconds of apparent issue out of 180,000 seconds frankly doesn’t surprise me at all. Is it unfortunate, especially for impacted teams? Absolutely. But, there’s people on the forum who are starting to act like it happens every match, and are also implying that VEX doesn’t care or isn’t doing anything to prevent it. I firmly believe those things to be false; I know very well how hard VEX works to ensure that things work. The reality is it will never be perfect. Sorry, that’s just life. And, that’s where we get back to one of the big problems here: winning isn’t supposed to be everything in VEX. Every student involved in VEX is having this amazing experience, and learning life lessons that will last a long time. VEX works hard to make their equipment work the best that it can, and they are constantly working always to make it better, but at a certain point the answer really becomes “we’ve done all we could, and it’s not supposed to be only about winning anyway”.

TL;DR: Wireless is tricky and hard to both visualize and diagnose. ANY wireless transmitter could possibly interfere with any other wireless receiver. An easy way to probably reduce the risk of interference in your match is to keep unnecessary transmitters away from the field.

P.S. Obviously something went wrong in that match that I responded to. I personally do not think what happened there was caused by any team’s robot equipment (i.e. IMO it wasn’t any team’s fault, unless they were actively using a WiFi device against event rules). However, if that happened at my event, I would NOT replay the match, because it didn’t appear to affect the outcome. Perhaps that was the judgement made by the field staff at the time as well.

@Dave Flowerday You are correct. It was not a half a dozen. You only mentioned 3 in that one post and a 4th way in another as far as mentioning things that could interfere with VexNet in such as way that it might cause disconnection in one, two, three, or four robots. In addition to the ways you specifically mentioned (all of which are partially out of the control of the team and some which are completely out of control of the team) there are other factors that could cause an issue for an alliance or the whole field.

The point of all this is not to cast blame at Vex or to say they are bad people, but rather, to find solutions to make things fair and equitable. If there is a disconnection that is as likely or not to be something that was not completely in the control of the team, then maybe, while we are using this system, RECF/Vex/GDC/Whoever makes these policy decisions should consider allowing for replays. If it is a situation where multiple robots are affected at the exact same time, then it seems like a replay would be a no brainer.

1 Like

Find me a team that hasn’t had a robot lose connection in a match.

1 Like

Only one of my 14 teams had connection problems during a match this year, and that was due to a crossed pin in their joystick. Beyond that we haven’t had VEXNet problems during a match. Only outside of it.

2 Likes

both of mine have had issues. Only one during elims.

I had the issue when i was in HS, but that was back when we were slapping the vex net add on to pics, those were worse than the cortex.

I work for a company that consistently pushes quality and customer voice. One of their favorite quotes from the founder goes like this.

“We have to aim for 120% product quality. If 99% of the products we make are perfect, that
would seem like a pretty good record. However, the customers who become the owners of the
remaining 1% will surely consider their products 100% defective. It is unacceptable that even one
customer in a thousand–even one customer in ten thousand–should receive a defective product.
That’s why we have to aim for 120%.”

saying “sorry it happens deal with it” is unacceptable to me, and when this many customers are voicing opinions doubling down on how we are successful 99% of the time doesn’t jive with me.