Disconnect Issue

At our competition today, the robot ran fine in the first two matches. However, in the following two matches our robot disconnected from the controller and was unable to reconnect. The motors kept running at low power after it disconnected. Both times this happened about 10 seconds into the driver control period (autonomous has run fine). When the robot disconnected the Vexnet light on the Cortex flashes slowly and the robot light flashes very briefly and then waits flashes briefly and waits. (Both are Red) (I hope this makes sense). On the controller, all lights are green except for the robot light which is off. We tried cycling power on the controller however, it made no difference; the lights on the controller still go all green and robot light stays off.
I assume this is something more complex than the regular disconnect issues, and so I attached the code.
vexuser.c (18.4 KB)

Hey, are you guys a team that went to the New Jersey Ranney competition? Regardless, I can relate exactly to your situation. At our first two competitions this season we weren’t able to do anything cause of connection issues. (tbh, I didn’t read your code, but I doubt there’s anything that can cause disconnection in your code).

We eventually solved our problem by getting brand-new, factory-fresh VexNets. I don’t know how VexNets “degrade” but I guess they do. Or it might be because they work better in their original pairs, although this does not make sense either. Or maybe Vex made a slight upgrade to the VexNets v1 recently… I don’t know.

Another thing that helps is having no WiFi. When using unreliable VexNets, I’ve noticed that they become substantially more reliable when they are used in WiFi–free areas (that’s why there is no WiFi at worlds).

However, because Vex is supposed to release new VexNets this month (would you guys at Vex mind saying exactly when?) which use a more free band of 2.4GHZ, they should help out with connection issues… a lot. If you don’t have any competitions until the end of the month, I’d order the improved VexNets with fast shipping when they are released. But then again these may not work neither… and that would have resolved nothing.

The slow flashing red led means that the master cpu stopped communicating with the user cpu, when that happens motors will continue to run at the last commanded speed.

I see you are using ConVEX, there are at least two significant issues with the user control code, SmartMotorRun() is called within the main “while” loop and there is no call to vexSleep (it’s been moved outside the loop). I’m sure multiple calls to SmartMotorRun will be a problem but let me analyze the program properly later and get you a fixed version.

ConVEX (and PROS) cannot protect you from coding errors in the same way that ROBOTC can. Bad code (and by that I just mean code with certain types of errors) will crash the processor.

I reread you post.

The VEXnet led flashing red is a disconnect issue, and has caught out some world class teams, that’s what happened to Telemascope in the 2013 finals. Not sure what caused that, I cannot say for certain if the code had anything to do with this or not (although I would probably say not).

I looked over the code, I could not reproduce your issue (using the latest build of ConVEX) but there are several small issues that need addressing, the highlights I will hit here.

SmartMotorRun() should only be called once. I looked at the code for this and the ROBOTC style startTask function it calls is smart enough to ignore all but the first call, however, that was not in the very first release and it is generally bad practice. You should start it once in the vexUserInit() function.

You should (must) have a vexSleep in the main while loop of both auton and operator control functions, the code will run without, but some tasks (shell for example) will never get any cpu time as they are lower priority and some tasks (lcd) will not run very often as they are the same priority.

The code to control the pneumatics was screwed up, you had left the demo code that controlled an led on output 1 running, you don’t need to constantly set the port to be an output.

The LCD select had trouble selecting an autonomous routine, I added a check for button release to solve this.

For clarity, keep all the “#define” declarations outside of the function, a couple of these are redefined, I made comments about this in the attached source.

I also made several other changes to remove all warnings from the compiler, mostly about unused variables, try and have code compile with no warnings as sometimes they are relevant to code functionality.

I was going to replace all the “magic” numbers (motor ports etc.) with their enumerated types, but decided that that was going too far and I may make a mistake.

So here is a revised version.
vexuser.c (21.6 KB)

Oh, guess I was wrong about my assumption that the code was fine. Sorry. :o

The code was most likely not the original problem, however, the code did have some issues.

It sounds like the key or battery (you make it sound more likely this) gets slightly dislodged when a dropout occurs. Does unplugging and plugging back in the key and battery resolve the issue? Pertinently, what do you do in order to “fix” the issue, even if it doesn’t hold out.

Here are some things I’d do:

I’ve found that loose battery connections and/or VexNet keys are often the source of issue. Even when connections seem secure, they may not be. Try gently wiggling or flicking the VexNet and battery connections - obviously not enough to break them but enough to see if they are loose and end up causing a dropout. Or better yet, give your robot the shake test. I like to make sure that I’m not buying unnecessary parts, so perhaps also simulating a match on field while practicing would help. Drive like you normally do - rigorously and perhaps even a bit recklessly in order to help determine what conditions make it dropout. Obviously, everything I’ve said should be taken with a note of caution: don’t break your robot. Push it to it’s tolerance limits but don’t go past them.

Also, if you have a USB extension cable, get rid of it to see if it helps. The USB ports on the cortex have the little rails to help secure the key and I’ve found it’s way too easy to get a loose connection on the cable as we like to buy the cheaper options.

One of my predictions for the dropouts is lightly loose connections get broken when jostled and when the battery is under a high voltage load, I.e. lift it or bumping walls. Just throwing that out there as food for thought.

Being a ConVEX user, I’m sure you keep firmware up to date, but I would also try downloading firmware (even if it’s up to date), including the slave and master code.

I know just how frustrating it can be to have a hard to reproduce dropout issue and eliminate it. I have random things that I’ve tried before that sometimes seem to work, but are by no means tried and true. Then again, desperate times call for desperate means, so I can share a few desperate means that should have little effect, but ended up doing something.

Hope this helps!

No, we’re not.

We are planning on getting the new keys when they come out and in the meantime I’ll see if we can RMA some of ours. However, we have a bunch of competitions in quick succession for the next couple months, and I don’t think we have enough extras to send some in.

Darn, I liked your first explanation better.

When the controller and Cortex loose connection, shouldn’t they be able to reconnect without cycling power on the Cortex?

We have done testing like this in the past as well as at the tournament and found a few batteries as well as keys that didn’t like being jiggled. We were sure not to use those. Although, it’s certainly possible that this was the problem. Our key is taped in and we did try wiggling both with no effect. Also, the third consecutive time that we lost connection the robot wasn’t even moving.

We had to cycle power on the Cortex in order to make it reconnect. I suppose unplugging the battery and plugging it back in would also allow it to reconnect (might have to unplug backup battery too? I’m not sure).

Also, we were using the same VEXnet keys the night before as well as the morning of the competition while we practiced driving and had no issues until our third match. After we lost connection the second time, we switched to a new pair of keys to no avail. This seems to happen about 15 seconds into driver control no matter what we’re doing.

I’ll update the firmware this week.

I know the whole VEXnet connection is rather fickle and so unfortunately as you said it’s hard to troubleshoot. Hah, and I thought we had orked out all these kind of problems before this tournament. Thank you, everyone for your input, especially Mr. Pearman for rewriting my code. Before I change anything I’ll see if I can reproduce the issue at home. It happened three times in a row (once it started it happened every match), so one would think reproducing it would be easy. But we all know how it goes . . . it won’t be.

We had this happen to us at two tournaments, finally we found that the problem was bad controller batteries that when he remote was slightly shook, or used from extended amounts of time, it disconnected. Not sure if that’s what you have but we had that happen to us.

On Monday, we drove the robot under similar conditions as at the competition (plugged into a competition switch, ran the same auton program and then switched to driver, same joystick controllers, same batteries in those controllers, same VEXnet keys, and same batteries on the robot). Surprisingly, it lost connection every time very quickly into driver control. We tried switching the VEXnet keys to ones that worked fine for one of our other teams at the competitions and also put fresh batteries in the controllers. It still consistently disconnected every time (about 5 trials in total yesterday). Today, however, we have not been able to reproduce the problem at all. Nothing changed on the robot since we left on Monday and we have followed the same procedure. We ran a total of 5 mock-matches today and no problems occurred. I have tried wiggling the keys in both the Cortex and the controller, wiggling the battery cable and connector, and shaking the robot/controllers, and it hasn’t disconnected once. We fear that the problem still exists and is only lying in wait for our next competition.

What sort of interesting ideas do you have to try to make it appear again? I’ll drive the robot for more extended periods tomorrow and we’ll see what happens.

We are losing communication consistently with Vexnet. At this point I am only running the sample 4 motor code from the RobotC library. The robot works fine tethered with USB cable. As soon as I connect with Vexnet, I have all green lights on joystick and robot. Game is not lit. As soon as I move the robot a few inches, immediately the cortex flashes: The game light flashes very fast yellow on the cortex. The vexnet on cortex is flashing orange and much slower. The robot light on the cortex flashes slow green. The joystick light stays green. The vexnet light on the joystick flashes red slowly. A few seconds later it restablishes connection and as soon as I touch the joystick, it loses connection. After I do this a couple of times, the cortex vexnet light flashes red slowly and it never reestablishes the connection. It almost seems as if the motors turning on causes the robot to lose power. I am using the 9V backup battery. The battery in the cortex is fresh. All firmware is new using version 3.62 of software.

I am not a programmer. This is our first year and I have no idea where to go from here. Any assistance is greatly appreciated.

One more thing, here’s a video of the lights on the Cortex after it disconnected Monday. Based on this:

Game light quickly flashes yellow = disabled
VEXnet light quickly flashes green = linked
Robot light slowly? flashes red = fault processor issue

Is this correct and if so what could cause a microprocessor error?

Edit: here’s the video
http://youtu.be/NSCGgxPLmUU

Are you running the code that I revised for you?

What version of ConVEX are you linking against? (either look in vex.h or on the LCD when it boots).

No, I hadn’t uploaded your revised code yet because we wanted to change as little as possible while trying to reproduce the problem.

Version 1.0.2

Well, there’s nothing that should crash the user cpu in any of this code “theoretically”. I will wait and look at your video when it’s up and linked. One possible problem, did you have a backup battery connected ? When on the competition switch the lack of backup battery causing the robot light to flash red masks other problems.

See here for an example of cpu crash leds (post #2), it’s a short flash not a long flash.

https://vexforum.com/showthread.php?p=326053#post326053

The video is finally up http://youtu.be/NSCGgxPLmUU and it looks just like the graphic in that second post.

Yes, we had a backup battery.

It does look like a user cpu crash.

I will look at you code again later as I notice that there is a major conflict between your use of vexMotorSet and the smartMotorLibrary. Normally vexMotorSet is used when the smart motor library is not included, when it is then SetMotor is used in place of vexMotorSet (for historical reasons to be compatible with the ROBOTC code). I’m surprised the code worked at all and my revised version is probably even worse than the original for motor control.

Ok, we were originally using the smart motor library and had some problems with the motors losing power too quickly. Rather than troubleshoot it, I just took out some of the smart motor code (changed the SetMotors to vexMotorSet and removed the ptcMonitorEnable()) and left some other parts (such as the initialization, link motors, etc). That probably wasn’t the best idea in hindsight . . .

The main thing we need to remove is SmartMotorRun, that was one of the things I pulled out of your driver code (it was being called multiple times) and moved to vexUserInit.

When you were driving on Monday did you notice any problems with static? Any sparks when you touched the robot or the robot touched the field perimeter? This was a root cause of some of last years problems, especially on robots using IMEs and mecanum drive (which it looks like yours may be).

The static discharge would cause the user processor to do something crazy, exactly what we never really knew, but communication with the master processor would be lost and the program stopped presumably due to an un-handled exception (bad memory access, whatever).

I know that here in socal we have days where the static is really bad, sometimes in my lab everything I touch causes a discharge, that was happening earlier this week but I did not notice that today.

Part of the fix we came up with in ROBOTC was implementing the watchdog timer, this will reset the user cpu if the code goes crazy as I described above, I will show you how to enable this in your program as it is a feature of ConVEX that is normally disabled (I don’t really like them).

I ran your original code again and, although there are the issues I described before, it did not crash for me. I cannot exactly duplicate your setup (I could but I’m overloaded with work and don’t have time), but nothing like a stack overflow or anything seems to be happening. The original code should have had more problems, however, because you omitted the vexSleep call in the main driver control while loop, the smartMotorLibrary really didn’t have time to overwrite the motor values you were setting, that was just lucky I guess. The revised code I previously posted is probably undrivable due to the smartMotorLibrary conflict ( I didn’t test with motors connected, I just checked values in the shell ).

I will post yet another revised version with smart motor turned off for you as well as watchdog timer code.

Edit:
I did find a way to crash the code although probably not your problem in the competition. Running autonomous twice without a reboot was bad, you start the gyro task in auton and calling vexGyroInit() twice is bad. Once the gyro task is running it should not be restarted, call vexGyroInit() in vexUserInit() one time only.