Thoughts on the new elimination rounds system?

Hey now. I made the metaphor, because it was the best one available for a discussion purely about statistics. Please don’t call him out for it.

My apologies. I get the point of the metaphor, although I do think that logic is flawed in the fact that when laboratories test on mice, it’s with the intent to actually roll out that product to humans over time. In this case, you’d be testing out a product on other humans, not [the equivalent of] mice. That’s the big difference here, and in my opinion that’s why the idea of testing this out at worlds is absurd.

1 - Regarding introducing the new format at VEX Worlds with little notice:

This is not the first time that VEX has introduced something new at Worlds. This is not a comprehensive list, but to give you an idea:

  • Raspberry Pi’s for field control
  • Tablets for scoring
  • Programming Skills Challenge rule changes (Gateway)
  • On-field sizing tool
  • Fields on risers
  • PTC tests (both in inspection, and post-match)
  • 3-field divisions instead of 2
  • Various VEX U elimination bracket iterations, including a multi-tiered one with 20 teams, single elimination for seeds 13-20, various levels of byes for ranks 1-12
  • The Round Robin elimination format
  • The concept of “divisions” and “division finals” at all

Worlds is the perfect place to insert these types of changes. We have the skilled event staff prepared to manage them, the skilled teams prepared to weather them, and the opportunity for immediate first-hand feedback. We would not want to roll something out to all Event Partners without first running it at the event where VEX and the REC Foundation are, in essence, the “Event Partner” ourselves. If it’s not good enough for Worlds, why would we ask it of the 1500+ events that occur around the globe each year?

Furthermore, I will admit that many of the changes above were made without proper advance notification to teams. Increased transparency is something that is very important to many of us here, which is why we felt it worthwhile to notify teams inside of the scheduled April 5th update. This update cannot feasibly be scheduled much earlier in the season, as State/National Championships are still running through March.

2 - Regarding the concept of “luck”:

Many have commented that this system introduces an element of luck. I think that there is a key difference between “introducing luck” and “reducing the allowance for bad luck”. First, make no mistake - there is an element of luck in all forms of competitive robotics. The concept of a randomized qualification schedule, where you only play 6-12 matches, is extremely ridden with the opportunity for good or bad luck. But, because that’s the system we’re all used to, few people bat an eye at it.

A 16-alliance best-of-1 system does not introduce any new elements of luck. “Getting lucky” in the round of 16, and the quarters, and the semis, and the finals, and the Round Robin (for HS), and the Freedom Hall finals… it’s just extremely unlikely. A team cannot trip and fall their way into winning an event, just like they couldn’t before under a best-of-3 system. I have full confidence that the teams representing their divisions in the Round Robin and the Freedom Hall finals will be some of the top teams in the world.

Now, does it remove some tolerance for bad luck? Sure. But this is the case for the qualification ranking system, too. And just like the qualification ranking system doesn’t always result in the “best” robot seeding first, there is a possibility that the “best” robot may not win the event. This has always been the case, it is why things like the Skills Challenge or the Excellence Award exist, and why the Excellence Award is considered a higher honor than Tournament Champion.

3 - Regarding “VEXnet disconnects” being the primary perpetrator of bad luck:

Every time that someone says “of course VEXnet is terrible and disconnects randomly”, a VEX field tech twitches. “Random disconnects” just do not happen, especially once VEXnet 2.0 was introduced. There is always a reason. Sometimes the reason is low battery, sometimes it’s a loose physical connection, sometimes it’s a bent pin in the competition port, sometimes it’s the fact that the VEXnet Key is buried under a giant steel mechanism that looks impressive but does a great job of shielding wireless signals. Sometimes, in very specific and identifiable cases, the reason is a field issue.

Some of these root causes are more readily apparent than others, admittedly. Blink patterns are an outdated way of communicating this information, admittedly. This is what leads to the “random disconnect” sentiment. But, that is why we employ the field tech crew at Worlds. Every division has a Senior Field Tech who is a bona fide, real-life VEX engineer and two other Field Techs to assist them. There is a Software Support center near the Inspection area with more VEX staff. The sole responsibility at the entire event, for all of these people, is to help teams troubleshoot and identify these root causes.

4 - Regarding this “being done for TV”:

I am trying to keep my comments broad rather than answering specific posts, but this is one that I want to nip in the bud right away. As part of the production crew at Worlds, in addition to being on the GDC, I can assure you that we would never sacrifice team experience or competition integrity solely for the sake of television. Audience perception plays a role in game design and event structure, absolutely. But, this pertains more to things like making sure that an audience can quickly ascertain what is going on when they see a game being played for the first time. “Best-of-1” is definitely easier for an audience to understand. But, that’s a benefit that came after the decision was made, not the determining factor.

To be perfectly clear - if ESPN themselves came to us and said “you know, this would really look better if the game objects were bigger”, or “this alliance concept is really weird, can you just do 1v1 for this event?”, there would be no chance of them having that type of influence.

5 - Regarding the “overwhelmingly negative community response”:

This thread has done a great job bringing to light many of the concerns and complaints that this particular segment of the VEX community feels. This feedback is invaluable, and the heated response is a direct representation of your passion for the program. But, in addition to the sentiments being shared here, we (Dan, Paul, and myself) have also received just as many private messages in favor of the change. Beyond private messages, Paul had the opportunity to speak with a number of teams directly at the US Open this weekend, and I have had the chance to speak with a few teams here in the north Texas area.

It’s important to remember that when controversies like this arise, the people posting are going to be those who have a strong reaction, and a strong negative reaction is almost always more compelling to send than a strong positive one. Look at Amazon reviews - they will usually be biased towards 1- or 5-star. Because someone who buys a phone case and thinks it’s “okay” isn’t going to go through the effort of posting a review. My point being - one side of the story is being very well represented here, but I would be cautious to assume that this is a unanimous opinion.

Since so many others are giving their opinion, I will as well.

At this time with the un-reliability of the fields and electronics I think BO3 is a no brainer. I know Paul says the fields are accurate 99.5% of the time, but as an EP, Coach and Parent I have seen much lower percentage of error free matches during my years in VEX.

In fact, this past weekend at the US Open I witnessed (MS Division) many times that robots would randomly stop working. During the event the announcer was constantly telling the audience to turn of their hot spots, so he was obviously noticing the same issues I was. If hot spots are that detrimental to the connections then we need to figure out a better way to eliminate them.

Once the field electronics accuracy improves I am more than supportive of switching to a BO1 format. Until that accuracy improves, my students success and chances of a trophy could be more influenced by field issues than their talent, skills and hard work.

Oh, here comes the discord train coming at me lolol…
After giving thought and changing my mind countless times, I am now swaying to favor this “Best Of 1” thing. I believe that the “Best Of 1” idea will promote teams to be further prepared for every elimination match, and also be aware of every decision they make during a match, increasing the consistency of every team. After going through all of my competitions that have “Best Of 3,” I realized that I didn’t give much care nor preparation for the first match of each ladder during the elimination rounds. I seem to always have the mind-set of “If I fail, I always have another match I could do,” and I realized pretty quickly that I lose a match simply because of lack of care for the first match of an elimination round. A great amount of times with the “Best Of 3,” I lose a match the first time, to simply realize that I actually have to win TWO matches IN A ROW, in order to proceed to the next elimination round. This is one reason why I think the “Best of 1” idea is actually a good idea.

Now, here are some suggestions I would recommend, as someone favoring the “Best Of 1” Idea.
In order to make this “Best Of 1” Idea as fair as possible, I believe that VEX Worlds should allow teams to have more time to set up their robots on the field and to ensure that everything is where they need to be before every match. I’ve calculated that the “Best Of 1” matches with 16 teams will end quicker than the “Best of 3” matches with 8 teams. I would suggest that if there is a connection issue, or it is near assured that there is a connection issue, then I would think it would be best to do a rematch to ensure that the matches aren’t being lost just by a faulty connection to a computer. I would also like to suggest that time outs be 5 minutes instead of 3, or even have two time outs per alliance at worlds to ensure that teams are most prepared for the matches. I’d like to suggest as well that the World Championship should also be open to allowing all replays by users, OR do replays with their own cameras to ensure that disqualifications are legitimate.

@Grant Cox Thanks for your post Grant.

I have a few concerns regarding it.

I understand that there are field techs at worlds who are able to help teams address disconnection issues after the fact. I believe most teams concern is whether, in the case that a disconnection takes place that the team is NOT responsible for, the match will be replayed or not. Cases where a whole alliance disconnects at the same time are almost certainly due to field control (the chance of two robots on the same alliance disconnecting at the same time due to a loose battery or vexnet key is incredibly low). This happened in US Open F1, as well as a high school World final match last season. In neither of these cases was the match replayed. Can you confirm that you are releasing guidance to field techs and referees that matches in which field control causes disconnects at the World Championship will be replayed? I believe this would help to alleviate many teams concerns.

In addition to this, is there a foolproof way of identifying connection issues caused by field control in the VEXU competition, where we will obviously not see two robots disconnect at once? Perhaps you could consider running both teams’ robots off of the same field controller?

Thanks for your consideration,

I would like to add to @Telemascope 's post that that three of my 10 qualification matches at the US Open had simultaneous disconnections, two of which were for my alliance, and none of which were replayed. Instead the refs pointed to the green driver control light on the field control and said it was our fault.

Let’s just make this simple. If there wasn’t such thing as issues with field connections, VEXNet keys, as well as issues with battery connections and teams are given a nice amount of time to prepare for the matches, would most people be neutral or supporting of the “Best Of 1” Idea?

@Grant Cox

Thank you for your response. It’s this type of back-and-forth that creates the ideal dialogue needed to properly debate a topic such as this. Allow me to likewise address a few things about these points:

  1. The first 7 things on that list have absolutely nothing to do with this topic. Those things are either minor technical changes meant to make things easier for event staff and competitors, with the exception of the Gateway rule change, which I can’t really comment on because I wasn’t a part of VRC at the time. The last 3 bullets actually pertain to a major bracket changes like this, so those are what I will focus on.

In regards to the VEX U brackets, you’re using their situation as a means to justify doing the same thing to us (and honestly, it’s not really fair to them, either). Is it okay to change a child’s footwear from sneakers to roller blades and just expect them to adapt just because their older brother is adept at wearing multiple types of footwear, sneakers and roller blades inclusive? I realize this may not be the best analogy, but surely you get the point. Not only is it questionable for you to do this to VEX U to begin with , but using an already questionable system to justify a second questionable system doesn’t really do the second system any justice. If anything, it’ll just lower its standards by association with the first questionable system.

For the implementation of a round robin elimination format, this was actually a change for the better representation of the most skilled team. With round robin, each alliance faces every other alliance in a best of 3. Because every single alliance faces each other, there is a balanced representation of who is really the best, as no arguments like “Team X and team Y never faced each other during elimination rounds, so even though team X won the event they could’ve easily been knocked out by team Y.” can be made. Meanwhile, this change to a BO1 format with even more alliances does little, if not less than the previous system, to actually determine the best team. If anything, it does that even less due to the inherent luck factor, which I’ll discuss in my response to your second point.

I’ll skip over divisions and division finals because it pretty much gets the same explanation as the above paragraph.

  1. Let’s not get too technical about what specific type of luck is introduced or reduced, because we can all agree that in the end, the net amount of luck needed in a BO3 format is less than that needed for a BO1 format, regardless of whether or not it’s “good” or “bad” luck. In regards to a randomized qualification system, people actually do “bat an eye” at it; in fact, it’s been a large discussion for a matter of years. Several more accurate systems exist, such as Swiss (used in many popular gaming tournaments) format where rounds aren’t determined at the start of the day, but are determined after each current round ends, and teams with the same W/L record are paired against each other in order to weed out the undefeated teams one by one. Because going undefeated in Swiss events requires beating every other undefeated team, it really does a great job at not only pairing the best with the best, but gives smaller, less experienced teams a chance by pairing them with other teams at their skill level. So yes, that’s also an issue, but a completely different one that the GDC has not yet addressed.

Regarding having to get lucky all through R16 to the finals being extremely unlikely, that’s not how that works. Yes, the probability of one single team having amazing luck throughout every single match is low, but what I think you’re missing here is that we’re looking at 32 teams, not just one. The probability of at least one team having good or bad luck in at least one match in each R16, quarter-, semi-, and finals matches is actually pretty high if you consider the pure amount of teams competing. This, through a butterfly effect, skews the overall results of the competition. If seed 1 had a field control error in R16 and lost, it means that seed 16 will now move on to quarter-finals, so whoever faces seed 16 will have a much easier time than they did if seed 1 had won and no bad luck was had (which is much more likely to occur in a BO3 system). The point is, things add up. Bad luck adds up. Not just on one team, but on 32 teams as a collective.

  1. Every time a ref rules that no replay is needed on a match where VEXnet disconnected randomly, every audience member, drive team member, and live stream spectator does a lot more than just twitch. Yes, there is always a reason, but the chances of those reasons being caused by both robots at the exact same time, for the exact same amount of time is so ridiculously low, I’d argue that I’d have a better chance getting through R16 to finals on luck alone. Yet, this has happened twice on camera, both in F1 of Worlds 2017 and in F1 of this year’s CREATE U.S. Open. Please don’t pin situations like this on battery voltage or loose pins, because if it happens to both teams on one alliance, clearly it was something with the field control itself, and it’s these situations that we’re talking about.

Thank you for providing a field tech crew at Worlds, but clearly that didn’t work so well last year as the aforementioned finals match was not replayed even after something as blatantly obvious as that occurred. I’m not sure if the field techs were involved in that so much as the refs, but there are some very clear issues that need to be fixed here if this point is to actually be valid.

  1. I don’t really take any issue with this, so thank you for the assurance that your players are your number one concern. Hopefully this comes true not just in words, but in practice.

  2. Although I don’t really believe that there have been over 150 positive messages sent about this change as this thread has received replies in dissent of the change, I’ll give you the benefit of the doubt on that one. I’d just like to explain how opinions like these given through the forums are actually the most accurate opinions you’ll ever receive:

In person, people like to appeal to their audience. They like to know that they’re being listened to and they want to please who they’re talking to. I notice you didn’t mention the opinions of the people you and Paul talked to at the Open and in Texas, but let’s just assume they weren’t all negative. Those opinions were given with the bias of wanting to please a senior representative of VEX robotics. Obviously they aren’t going to say much negative, as they’re talking to someone extremely important! On the flip side, here on the forums, people can use the virtual mask of the Internet to provide the most honest opinion possible. Sure, I may get to meet you guys at Worlds, but right now, I don’t have any obligation to please you with my speech. This is why I can be as honest as possible, because there is no bias towards an elder here. It’s just pure, candid, honest speech. I can say the same for most of the other posts on this thread, or on any of the other countless threads that all echo our same message.

So, much like you say that we should be careful in assuming our opinion is unanimous (which we don’t), I urge you to be careful in assuming the opinions you’ve received in person are actually the opinions they hold when they aren’t speaking to someone as important, and are a little more comfortable sharing their real opinions.

Thank you for reading this (if you do), and I hope this echoes some of our concerns as a community towards these points. Hopefully our civil discussion can continue on in a cohesive fashion as it has so far.

@[TVA]Connor I would be less against it, but still against it as I believe that a single 2:00 match is not the best way to determine which set of robots is better.

For “Best Of 1,” since 3 minute time outs will probably be used by every team since teams know that they need to be prepared, think about this:
Why don’t we remove the 3 minute time outs entirely, at worlds eliminations, and instead introduce a Challenge Match where the higher seeded alliance in a branch is allowed to do ONE rematch during eliminations if they’ve lost to see if they are actually better than the other alliance. If the alliance that provokes a Challenge Match wins, they will be allowed to do a tie breaker match to be able to get back into the branch.
I think that this Challenge Match should only be allowed to be used by the higher seed only to ensure that there is efficiency on time as well.
By the way, I think the analogy of the “Challenge Match” should be called the “Challenge Match” because the higher seed is challenging the loss of their previous match.

Now, If you believe this is a bad idea, this also benefits the lower seeds as well. If I were in 16th seed, and I’m against the 1st seed. I win against them once, and then I win against the Challenge Match as well, I would become the higher seed for the remainder of the eliminations and I would be allowed to do one Challenge Match myself. This benefits everyone, and yet it is efficient on time too.

1 Like

Believe me, it almost certainly isn’t.

What is it then? I’ve been witnessing these simultaneous disconnects for four years now and am still baffled as to what causes them. The field control is the only common factor that I see every time an alliance simultaneously disconnects.

I don’t want to either derail this thread or get into a deep technical discussion here, but take a look at this;

This is a nice idea, I think that it needs to only be available during and after the quarter finals to reduce the length of the round of 16 and also it should only be available for a team to use once during the elims again to keep the length of the elimination rounds short.

The robots that go to the Round Robin, I believe, will be deemed most consistent in the competition. To be put through all that work and stress yet still function very well is a difficult task to pull off.

@Grant Cox Thanks for taking the time to discuss this on the forum. I don’t think anything any of us say that is contrary to what has been introduced on the 5th will be implemented, or, honestly, considered. Vex seems pretty set in how this will all happen. It is a shame, but the “our way or the highway” mentality is prevalent in a number of responses both in this thread and others.

I’m not sure where you guys keep getting the statistics that you post, but they don’t jive with reality at all. Either that, or every team I have surrounded myself with as friends and colleagues are the most unlucky group. For example, if the vex net system has a 99.5% uptime then, on average, a team should be able to go 200 tournament matches with only a single disconnect. For the more competitive teams that attend four or five tournaments, attend States, and make eliminations at each, that would be an entire season with just one disconnect. For teams that are in the middle of the pack that would be a few seasons with just one disconnect. Or, going by tournament, it would be a few good sized tournaments without a single disconnect. I would challenge you to find teams with that kind of experience. I am really impressed with the V5, the thoughtfulness of it, the features, lots of great stuff, probably some glitches and such getting worked out now. You guys seem to have done a great job with it, but, to the contrary, the vex net system has a lot of flaws. One thing I see a lot is defending the design and saying there are a lot of ways teams can mess it up (which is true). However, you guys designed it, if you designed it so it was that easy to have issues with, then own it and give the teams consideration when it comes time.

With respect to the latter, I bought up our terrible experience in IQ last year during one of these threads and Paul mentioned that it had nothing to do with the present discussion about one and done. I would contend that it does and that it speaks to the need for vex to start considering these kids more and the World schedule a bit less, or at least re-work it’s priorities. Teams are concerned about losing in the one and done for issues out of their control. I would agree. Based upon our experience last year in IQ I spent months emailing Vex that there was an issue with their boot loader, that it would randomly halt the brain and could happen at any power up. It happened at our State tournament and I emailed Vex again to let them know and asked how it would be handled at Worlds. You guys gave the same answer you posted, that there would be vex technicians on site to fix it and all would be well. Well… it happened to us at Worlds, your technician gave us a new sensor, and then made us use the placebo robot because they would not let us take the 60 secs need to implement it (back to stats, Paul stated IQ had 0.4% placebo usage so it was dropped, somehow we had 30% all on our own at Worlds last year?). It really upset our children and then I am the one getting scolded for being unprofessional and a bad example for our kids? Teams are afraid of the one and done issues not because of assumptions, but because of poor experiences with Vexnet, inconsistent ref’ing, etc… they have learned it, not assumed it. The good news is that there are opportunities to improve it, hopefully that will happen at Worlds this year.

This is a larger-scale, though very much less severe, version of the issue we had in our state for many years. Our RECF reps focused on getting tournaments done early, to the point where they said we could not send a team member back for batteries between elimination matches. Yes, really. Apparently the risk that somehow sending one kid back for batteries after an elimination match, with ten minutes of free time before the next match would delay that match from happening was not worth letting us have charged batteries. Nothing nearly as bad is happening here, to be clear, but this is seeming a little too similar for my comfort.

Got to agree with @TriDragon, since the manual update have been officially published, GDC isn’t going to change anything for the upcoming Worlds, with the exception of maybe bringing VEX U division finals to the Arena. It would be a huge surprise even if division finals are changed to BO3 based on the popular demand…

So, here is a crazy idea that might excite those of us who are scientifically minded, and think all experiments are supposed to be run with at least one control group and in the identical environments. Ideally, such experiments would have to be with randomly selected groups, but this thread is the proof that random selection is not going to go well with this crowd - we would have to live with self-selection bias. :slight_smile:

The experiment:

Upon arrival to VEX Worlds 2018 each High School team will be asked their preference between:

  • BO3 eliminations with 8 x 3 team alliances
  • BO1 eliminations with 16 x 2 team alliances
  • or choose “no preference” if they don’t know or don’t care

Then teams will be assigned to the divisions run either as BO1 or as BO3 according to the team’s choice and “no preference” teams will be assigned to pad those divisions randomly.

After each division crowns their finalists, 3 team alliances will have a choice of which 2 teams are going to play in each RR game as well as the Finals, assuming they will make it that far.

If such experiment were to be conducted, first, you would learn how many people prefer each system.

Second, everybody will feel great because they made their own choice.

And, third, assuming divisions uniformly split between BO1 and BO3, we would see which system is better at selecting best robot, based on RR and Finals result.

Of course, it is not going to happen, but, I would encourage RECF to let some sociology major students to conduct pre- and post-worlds surveys of the random selection or, if possible, all teams to ask them:

  1. What would be their elimination bracket format preference?
  2. How high in ranking do they expect to get (or actually did)?
  3. How far in eliminations do they expect to get (or did)?

Then you could run simulation of virtual tournament with game predictions trained on actual World match results (i.e. Elo Ranking or some other model) and see how each tournament format plays out.

This would be the gold mine for anyone majoring in sociology or modelling and looking for masters or PhD thesis topic!

I really hope this one round thing doesn’t carry over to next year. IQ runs a similar way, and it wasn’t fun. There’s a lot of pressure on the drivers. And the risk of a malfunction is real. I didn’t get to nationals last year because a small malfunction that happened maybe once in the whole tournament happened twice in the finals round.