Ok, so a point of clarification should be had. Last year during worlds round robin/finals the field techs said that it was the fault of all three robots, due to us having orange controller batteries. How this would work i’m not sure, however the field controllers showed red for the entirety of the time we were disabled/disconnected. The field controllers seem to have some way of disabling robots, and not others, weather that is through static discharge or interference. @jpearman @[TVA]Connor this still doesnt inspire alot of confidence in the BO1 system, where the best equipment in the arena cannot cope. To answer your question, the go pro on the ladder was turned into wifi off mode, and there were no detectable wifi signals as they checked that before the match started.
I really don’t see how this rules out “anything related to the field control system”
While it does probably rule out the field control system spontaneously disabling robots (though that is not at all the focus of our arguments; when people are complaining of disconnects they are usually referring to an actual disconnect, not just “my robot stopped moving”), it does nothing to disprove the possibility of surges or other electrical faults.
As for your explaination that the robots just happened to be on the same channel, that doesn’t line up with the statistics (my experiences and observations made throughout my years of vex). The number of times that two robots on the same alliance disconnect vastly outnumbers the number of times that two robots on opposite alliances disconnect, when if your guess is correct it should be 2:1 the opposite way.
This leads me to believe that the problem lies somewhere other than wireless interference for the vast majority of simultaneous disconnects. The only other common factor I can think of is the field control system.
My post was about a specific match with a specific video. I’m not addressing “people complaining about disconnects”, I’m addressing a specific circumstance in a specific video.
In that specific video, the behavior rules out field control. Go back and read the technical description at the top of this thread. The two teams’ joysticks on an alliance are linked by a direct electrical connection between the “disable” pin on each joystick. It is not electrically possible for anything in the field control system to send a disable signal to one team on an alliance and leave the other team enabled. That is the situation in this match; one red robot was not moving while the other was. There is no way for “the field” to do that.
My mistake. Do you have an explaination for the disconnects shown in videos such as the one I posted in this thread? That was my aim.
Also, as I stated in my other post in this thread, I am aware of how the field control works, and in fact have created my own system that emulates the function of the field control.
I would agree, if every part of the system is working as intended, it is impossible for 3 out of 4 to be disabled, but i also agree that if 3 out of 4 robots stop simultaneously, then there is a larger problem, probably out of the hands of the students.
I am curious what edge cases could cause this.
Would this behavior be possible if the one team who did not stop was the odd man out, and for whatever reason, the field control issued a stop to all robots, but the moving robot failed to obey that signal, either due to damage to a mechanical or electrical component, or something wrong in the competition template?
@Dave Flowerday I don’t think people here have an explicit goal to argue with anyone even tangentially associated with VEX. It would be sad if anyone did.
I believe, half of the controversy stems from the Paul’s 99.5% reply in another thread and how it was misinterpreted due to the lack of qualifying technical details.
He could have said: “Current control system is not perfect, but this is engineering and we are trying to improve it with each iteration. I understand your fears kids. At Worlds we will direct our best field technicians to stand behind every alliance in the division finals and RR. If there will be any problem they will help you to fix it or call for a replay.”
It is still not too late to assure everybody that field techs will be on their side and calm everyone down.
You are all aware that VEX already does this, right?
Yes, I know that, but fears of something that is both unknown and out of your control are mostly irrational.
That’s why the best way to combat those fears is to stay calm and keep reassuring people that somebody will stand behind their back and be ready to help at any time.
A field tech is no help at all to a well prepared team when all they are able to do is look at the joystick and field controller lights and tell the team that the field controller hasn’t disabled them, but rather they have disconnected (which they almost certainly already knew if they have any competition experience whatsoever). If my battery connection is solid, all my batteries are charged, my vexnet key is well located, my hardware is in good condition and has not been mistreated, and I am not using a wifi enabled device in the driver station, then what am I to do? If a field tech does not diagnose the issue as out of my control and request the match to be replayed, then there may as well not be a field tech in elimination matches, because no one cares to fix their vexnet issues after they’ve been eliminated.
No one was even suggesting that the field controller was spontaneously causing the robots to enter the disabled state, so why did this discussion come in this direction??? The proposal was that the field control could be causing the teams to DISCONNECT, not to enter the disabled state. This could be for any number of reasons, although my primary concerns relate to the length of wires used to transmit the field control signals, and static discharge. ESD on the i2c line of the cortex was already a huge problem in the past, and has effectively been fixed with the software equivalent of a piece of duct tape as far as I understand. Has anyone tested/is anyone able to test the response of the competition port on the joystick to ESD???
In my view no one has provided a satisfactory non field control related explanation for “full alliance” disconnections in this thread either. As far as I understand it the current possibility is extremely localised wifi interference. Given I vividly remember full alliance disconnections happening in toss up when both driver stations were situated on the same side of the field, this seems unlikely.
@jpearman and/or @Dave Flowerday
Along the lines of the question @blatwell asked in another thread, us Event Partners would actually love a primer on when matches could/should be rerun because they are likely a field fault. I guess I’ll just limit my question to this single scenario…
So if both teams from a single alliance lose connection at what seems like the same time, and then regain connection during the match, and nobody sees if the Driver’s Interface lights blinked to yellow, what can we reasonably assume is the fault?
We can look at the robots and see if they have low batteries on the cortex or joysticks.
We can inspect their joysticks to see if one of them has a bent wire in their competition port.
We can ask them if they shut off their joysticks.
Could the problem be the fault of a wifi device in only one of the alliance stations?
Could the problem be a drive team member bumping the RJ45 cable entering the bottom of the Driver’s Interface?
Could the problem be a referee stepping on a cable or cable bundle? (seems unlikely)
A poor connection to that particular alliance tower at the match controller could be the culprit? (unlikely again)
What could the likely causes be that would be attributed to the field and be cause for rerunning the match (in your estimation, I’m not looking for an official VEX answer).
Wow. Great thread, with some really, really good replies by jpearman and Dave Flowerday. Thanks, guys… where’s the “like” button?
The VEXNet system is actually really, really reliable. Robots do shut down and or reboot during matches from time to time. In lieu of information on the cause of the shut down, it is human nature for a team to assume that it wasn’t “their” fault. As a coach, I know that feeling… as an event organizer I have a less personally invested view on the topic.
Between VEX and FRC (and FVC, FLL and FTC) I’ve got about 15 years of experience with competitive robotics and have tried to pursue the cause of any unusual shut downs our robots have experienced (there haven’t been many, I’d say less than 10, but over 15 years they add up. That’s excluding the 2009 FTC control system which was truly horrendous. I just walked away from that nightmare.) A few I haven’t been able to figure out, most I have managed to track down to a minor flaw in code or wiring on the robot (our fault), and once… only once in 15 years… have I been able to find a reasonable path to suggest that it may have been a problem with the equipment or firmware that we have been provided. And that was on a 2009 FRC system, if I recall… not a VEX system. FIRST identified the potential flaw (not me) and addressed it for future years. LIke I say, this was potentially a field/firmware issue… I don’t have proof that it was. The rest of the failures I don’t even have a reasonable hypothesis to explain how it could have been a field or firmware issue.
In general my experience and understanding of the control systems involved has led me to take “field fault” claims with an air of healthy skepticism. It’s not impossible… I’ve experienced one in 15 years that I’m pretty sure wasn’t our team’s fault… but odds are it is some minor, tiny, difficult to trace issue with the robot or code that has caused the shutdown. One year it took us an entire afternoon and three lost matches (painful when you start the day undefeated) to figure out what was wrong with our robot and you can bet that the whole time we were wondering what was wrong with the field! We just didn’t give up on tracking it down. Eventually we found it and once again… it was us.
*I’m referring to unusual shutdowns in which the match was not stopped or replayed. In general my experience has been that if the field staff at any event have any reason to doubt the reliability of their equipment that they take proactive measures to address it. I am also not referring to cases of active interference with the control signals… something that I have seen documented once in a public forum. Once, and again… not with VEXNet. It’s not likely, but it is possible to jam radio signals. Theoretically the jam could happen from a freak unintentional source… but again, not likely.
No one is saying that robots are shutting down or rebooting… I can confirm this is not what’s happening because if it were then my robot would completely reset itself after every disconnect.
What is happening is a simultaneous disconnection of both robots on an alliance. There is no way that this is a problem with an individual robot or its code or firmware.
Please, please understand that we are not assuming that the problem is the field control. Instead, we have eliminated every other possibility, leaving nothing but the field control as the cause of these disconnects. Unless someone can come up with something else that can cause the problems we are seeing, we are going to ask for at least an acknowledgement of issues with the field control and measures that will be taken to protect teams from these issues, especially in light of the BO1 changes. As of now, we have been told that it cannot be the field control’s fault and that it must in some way be our responsibility, which is frustrating considering no one has told us what we are doing wrong that could simultaneously disconnect two teams.
I understand your concern, but I’m not the right person for that - any official EP guidance on when to re-run matches should come from the REC and/or VEX. It’s not my place to make recommendations on their behalf.
If no one looks at the driver interface LEDs, then I think you’re pretty much in the dark. Events really should have their volunteers/staff trained that in the case of any potential connection issues, the very first thing to look at is the driver station LEDs. The very next thing after that is the joystick “Game” LEDs, followed by the robot brain LEDs (assuming you can see them). Teams also should know this and rehearse it. It’s very frustrating to me when a team comes up to me after a match and says they had an issue but have no idea what was showing on their LEDs or on the driver interface. The other recommendation I have for teams is if they have an issue, alert the field tech immediately, so they can look at these things. Too often teams either don’t say anything until the match is done, or they yell at the refs. The refs aren’t going to be able to help, because that’s not their job, and they probably know less about the control system than the teams do. And if you don’t tell anyone until the match is done, you’re not going to get the benefit of the doubt, frankly. EPs see so many dead batteries and broken wires and abused equipment that they’re going to assume that’s the cause unless there’s fairly strong proof otherwise.
If you see green on the driver interface, then the “disable” signal is not being sent to the joysticks. From there, I usually look at the VEXnet LED on the joystick to see if there is a link. If not, then I try to get eyes on the robot’s LEDs. This is often where you can see that the robot code crashed, or the battery is nearly dead and the brain is rebooting (this is a general case, not specifically when both robots on an alliance stop moving).
Doing all this during the short duration of a match can be hard, especially when trying to reference the VEXnet documentation that shows all the combinations of LED patterns. At our events, we have a bunch of copies of that document and use it regularly for reference - being able to show a team having an issue a specific LED pattern and what that maps to in the documentation sometimes helps.
One thing that often catches teams off guard is the backup battery. From memory, I don’t think the LEDs show a low or dead backup battery unless you’re connected to field control. This causes many teams to show up at their match thinking they’re all set, only to see a non-green status before the match starts indicating a dead backup battery. I check for this before matches as much as possible. I warn teams who play with a low or dead backup battery that they are doing that at their own risk, and if they report connection issues we’re likely to assume it’s because of the backup battery.
If you DO see yellow/not-green on the driver station while a match is running, then you know something is very wrong. In every case that I’ve ever seen of this, it has always been a bad joystick - every time. Pins bent in the competition port, causing a short. That one gets real tricky too - lots of teams think they deserve a replay for that, but my stance is that if you bring faulty/poorly maintained equipment to the field, then that’s on you. Teams - treat your joysticks well. Unplug from field control carefully. Don’t rip the cord out. Inspect the ports on your joystick, and if anything isn’t 100% perfect, don’t use that joystick for a match. Borrow one if you have to. Frankly, I think the condition of the ports on the back of the joystick should be an inspection item to help with this. This is a problem that seems to be getting much worse in the past few years as most teams’ equipment is rather old now. Hopefully the V5 refresh helps there.
As I tried to explain earlier but perhaps failed, this (to me) falls in the category of “anything’s possible”, but probably not the most likely reason. My earlier posts on that topic were purely trying to say, “it could maybe be an issue, and on the off chance that it is, my advice is don’t take the risk of using a phone next to your drivers”.
Based on my knowledge of how the control system works, I would expect that any of these cases would result in the joystick simply becoming enabled (James explains this well at the beginning of the thread). That said, I’m not privy to the inner workings of the joystick firmware and what kind of corner cases or conditions (if any) could change that. It could be theoretically possible that a mostly-working CAT5 cable from the scoring table to the driver interface which occasionally has a continuity break because of people walking on it could cause the joystick to do something funky, like rebooting (which would appear as a loss of connectivity for a short period). Again, this is why paying close attention to the LEDs and knowing the boot-up patterns of the joystick can be really important.
The only thing I can do when I run events regarding these conditions are to try to prevent them in the first place, and swap things out if something looks strange. I bought a CAT5 tester (they’re cheap) that I check all our competition-related cables with before our event. We also replace cables after a few years of use (BTW - I believe VEX uses all new cables at Worlds every year, so that’s not a concern there). Nowadays, we use the Raspberry Pis on all our fields, which means we don’t have any field control signal cabling laying where anyone can walk on it (the field control cables just go from the monitor at the field to the driver station also at the field, so they stay on the perimeter).
At our last event, there were 2 teams in particular that kept coming up to me after their matches complaining that the fields were bad. We looked over everything very closely, including running their robots on a separate field control to test, etc. We never did isolate any reason (which I hate as much as they do). That said, whatever problem these 2 teams were having, it always followed the team, and occurred regardless of which field they were on or which alliance station (we keep notes on these things during the event). I have no doubt those teams left our event convinced that we had bad equipment which caused them to lose. But, all I can do from my perspective is to conclude that it is exceedingly unlikely that our fields would only happen to fail when that team played, and always on the specific port that team plugged into.
Now, despite what lots of people are reporting here, I have almost never seen both robots on an alliance fail at the same time. I don’t doubt it happens, but I just haven’t seen it much at all at my events. I don’t know if we’re just lucky or what. If it did clearly happen, and the match was close, I would certainly consider a replay. A lot of times, teams claim that their alliance is dead, but while one robot isn’t moving, the other one is partially working - arm works but not drive base, etc. I always point out that any controllable robot movement at all means that all the controls and communications are functioning, and that they’ve either tripped breakers or gotten a dead expander battery or whatever. And as usual, I don’t think they really believe me, because instinct is to assume it’s someone else’s fault. I know, I’ve been a robot competitor, and there were definitely matches that I felt we lost because of faults not our own - and I’m sure I appealed to the IFI staff who told me it wasn’t a field fault :/.
@Dave Flowerday Thank you for your response.
Here are some examples of simultaneous disconnects at the us open. 3 of these occurred in my matches, meaning that 30% of my matches were affected by simultaneous disconnects. I also suffered single disconnects but as those may have other causes I won’t post them here. If anyone else has videos of simultaneous disconnects, especially those with joystick, robot, or field control lights visible, please also post them, so that we may find similarities between them.
And here is a a match from last year:
@Dave Flowerday thanks for responding. My question: is the bolded section (emphasis mine) just what you would personally do at a local event, or what competitors can expect at worlds? I am not attending worlds, I just know that a lot of teams that are would be reassured greatly if the latter is true.
There is one failure mode that I have seen a number of times but have never been able to successfully recreate under controlled circumstances (as is the way with these things!)
During a match, you here the ba-dum noise that Windows makes when a USB device is unplugged, only the Match Controller is still plugged in and still powered up. Now the odd thing about this when it happens is it always fails in exactly the same way - one driver interface gets locked into enable mode and one gets locked into disable mode so one Alliance has a “simultaneous disconnect”. Only the aren’t disconnected but disabled.
I’m not suggesting this is what happens in the videos above because in my experience, the only way to resolve this issue has been to reconfigure the Match Controller using the TM system tray icon and it comes back to life, but you need to restart the match. In the thousands of matches I have run over the years, I have seen this probably 10 times. And, other than this odd failure mode, I have never seen a disconnect in a match that I haven’t been able to trace to team hardware issues.
These “simultaneous disconnects” as in the video above are not something I have ever seen but it always seems to happen to the red alliance in all of these videos. Is it always on the same field or different fields? 3 of the videos look like the same field.
You seem to have an unusual controller setup, do you have the partner controller mounted on the primary controller or is that just 3d printed paddles I can see?
It’s the partner joystick.
I posted pictures in this thread:
There’s no electrical modifications so I don’t see how it could cause problems.
It’s what I would consider at my own local event. I have absolutely no authority or input regarding what happens at Worlds regarding replays. VEX has the engineers who designed the system making those calls - no one knows how it works better than they do.
I suspect, though, that what “clearly happened” means to me is not what it means to a lot of you anyway.