Smart Field Controllers disconnecting with Tournament Manager

VEX V5 Technical Support
1

We just hosted our first Tournament one week ago and intermittently throughout the day the Smart Field Controllers lost connection with Tournament Manager. I was so disappointed because I really tried to follow all the instructions here: https://kb.roboticseducation.org/hc/en-us/articles/8497714209687-Smart-Field-Control

Our Setup:
3 Raspberry Pi 3Bs for competition fields
-Each with a 50 ft Ethernet connection to router
-Each with a 3 Amp power supply
-Each with a Smart Field controller that was purchased in the last year
-Latest firmware
-Updated the Raspberry Pi using TM on the Laptop (using the automatic function, not manually)
-Power cable running around one side of the field, Ethernet running around the other.
-Amazon basics USB Cable 3ft between Pi and Field Controller.

Power to Field Controllers was the Charger it came with.

Wired Ethernet to Laptop

We had about 20-30 disconnections in an 8 hour day. The teams would come to the field and it would not see the robots. Then we would unplug the USB from the Pi plug it back in and often times that would fix it. At the end of the day we often found we had to power cycle the Smart Field Controller and the Pi and then it would be running again. This seemed to happen more at the end of the day than the beginning of the day. There was a little more time between matches then with the elimination matches. Our head ref was the one who taught us that unplugging the USB from the Pi and then replugging it in would often fix it. He had apparently had this issue at past tournaments.
Our wireless Pis on the skills fields did not have this problem. Made me want to switch our main fields to wireless. Skills fields were run off of brains instead of Smart Field Controllers.

We have another tournament in February and I’d like to avoid these issues. Does anyone have any suggestions?

Also, I saw this post, but didn’t want to hijack it with my own issue: [Smart Field Controllers losing Connection with Tournament Manager]

2 Likes
2

Actually, these are related and probably should have been appended there.

I would recommend you look at the Event Log on your smart field controller to see what it detected along the day. Not sure it will but, it is one place where TM on rPi sends time stamped info to it.

1 Like
3

probably the first thing I would change.

The loss of communication I’ve seen before required more than just unplugging the USB cable.

2 Likes
4

Let me know if you figure out the issue. I’ll do the same.

1 Like
5

Thanks. I’ll try to get something even more robust. I had another event partner recommend these and they said they had never had issues.

6

Thanks for the suggestion. I’ll try to take a look there. Not sure how to get to the event logs but I’m sure there’s some information somewhere.

7

VEX has an extensive knowledge base you can search at kb.vex.com or you can look through library.vex.com.

Here is the kb article on Accessing V5 Brain Event Log:

It is a useful tool for teams, EP, and Head Referee. On team robots you can check errors that occurred on their robot. For V5 Smart Field Controller, it is the one log you can check after an event is over. During event, you can check details about each robot connected to the field.

2 Likes
8

I don’t know if you have fixed the problem but I run tm tech for 3 tournaments representing 2011 including Launch sig. If it is possible I would recommend using a POE switch to connect. We connect the router to whichever venue’s ethernet then connect the router to a POE and run all ethernet from there. The POE connects 2 computers and most of the time 10 raspberry pis with no disconnection issues. I’m not sure if this is a possible solution for you but our system has proven viable if it is a possibility for you.

9

I’m on my third year running my own events and lending equipment to others. While that does not make me a veteran or a pro at anything, here is my experience.

My setup is like what @Sauce described. A PoE++ router (overkill but there are reasons, I’m powering brains and other stuff from it as well) and everything else Raspberry Pis with a PoE hat from Waveshare.

I learned very quickly that the brain needs a very good USB connection and that length and quality are imperative. Today I use only high quality 1ft USB cables between brain and Pi. Long USB cables are an absolute no-no.

Next - and this is not your case but I’ll mention it - I started with Pi4 and was experiencing constant field disconnects. Through this forum, I noticed people mentioning that the Pi4 had some USB protocol issues (I don’t exactly remember details) so moved everything to Pi3B+ (at the time only 3B+ and 4 supported the PoE hats).

Ran 3 events without a hitch, system was rock solid.

Then I ran my own event and started getting field disconnects. Was so frustrated, ranted in the forum. @jpearman said something about someone running a microwave. I took that as a joke / conspiracy theory. But then I looked closely at the brain logs from all the fields (3 main 2 skills) and I realized interrupts were happening on all fields at roughly same time. The skills people simply did not notice while the match people were super vocal asking for replays.

Later I asked around and low and behold, the gym I ran the event in has an adjacent cafeteria where during the event people would go in and make microwaved popcorn. As usual, James knew what he was talking about…

Ever since then I carry a TinySA spectrum analyzer tuned to 2.4GHz to all events. Nerdy overkill? Perhaps. Peace of mind, for sure. And no more field disconnects, or at least I can show a screenshot and explain there was some major spike in the 2.4GHz spectrum that caused that and not TM, TM operator, corrupt people, bad cables and every other thing we like to blame this on.

TL;DR
Short and good quality USB cable.
Watch for 2.4GHz interference (WiFi hotspots and microwaves).
Give people guest WiFi because the millions of hotspots they start creating with their phones can be much worse. Banning WiFi makes them angry.

5 Likes
10

So this is not exactly a recommendation, but I will say the cable I use most in the office/lab is this one.

Monoprice Select Series USB-A to Micro B Cable, 2.4A, 22/30AWG, Black, 6ft
$3.39

https://www.monoprice.com/product?p_id=13926

I wouldn’t say it’s necessarily high quality, but I’ve been using that type for years with V5 and never had an issue. It’s also available in shorter lengths but I always find that too restricting with my setup.

I’ve done extended testing (100+ hours of running) with Pi3Bs and V5 using TM with no connection issues with that cable.

2 Likes
11

Which USB cable do you use?!?!?!?

12

I used to use cables made by Anker, just because I trust their quality. I’ve used them for years for charging phones and data transfer and they never failed me.

For my new field setup, since my pi is right near the brain now, I started using these: Amazon.com: SUNGUY 6 inch Short Micro USB Cable 3Pack, 0.5FT USB to Micro USB Nylon Braided Fast Charging and Data Transfer Cord for Samsung Galaxy S7 Edge S6, Power Bank, Android Phone, PS4 : Cell Phones & Accessories

1 Like
13

Thanks everyone for the replies. I haven’t looked at the logs yet but I will be helping with another tournament tomorrow and we will see how they’re set up goes.

We definitely had an adjacent microwave. It should have been through a few block walls and pretty far away but I’m interested to see if there’s a pattern there.

I wanted to run POE switches but I’m powering everything off these cables and we didn’t have the budget to buy it switches powerful enough to power all these.

We will get some shorter and perhaps shielded USB cables and test.

14

Update: Just used our equipment at a friend’s tournament and this is what I realized. Once again my problem is the micro USB cable. I’m using a 3-ft version from Amazon basics and I had it looped and zip tied tightly to my power cables and HDMI to try to make it one thing to carry around when tearing down and setting up. I’ll try to post a picture here.

We were having multiple disconnections. About every 10 -15 minutes. I cut the zip tie and tried to pull the micro USB cable away from the other cables. We only had three more disconnections the rest of the day. Each time just had to unplug the USB from the pie or field controller and it reconnected automatically.

Oddly enough we didn’t have much issues at all with the skills field which I did not cut the zip tie on. But not a lot of people were using skills today.

15

Looking at the event logs from the event today. I’m seeing a lot of logs where it says Disabled. I’m assuming that’s normal but I guess I don’t know.

Every so often there is a FC: Radio - R1 1

But then more rarely I see FC: Cable - R1 R2 B1 B2 3

Is there any documentation on the log codes?

The logs for my tournament where we had a lot of disconnections have simultaneous FC: Cable R2 1 and FC: Radio B2 1 error codes. As well as plenty of Radio codes that appear on their own.

16

Found the documentation on the log codes. I’m not convinced that these error codes are related to my disconnection issues since the FC Cable errors suggest intermittent issues with the cable to the controller and the radio error with the robot. I think my issues are with Field controller and PI and I think it is the micro USB.

17

Yea, those codes are related to team radio and cable disconnects. R1 means red team 1 etc. we unfortunately don’t have capability for very descriptive log entries, so we just use R and B as shortcut for red and blue. I don’t remember if we log usb cable disconnects, probably not as normal use of a V5 would encounter that often.

1 Like