Concerns With Rule T1b

VEX V5 Robotics Competition VRC > Tipping Point (21/22)
124

I know that currently white screens are grounds for a replay, so this is no longer an issue, but it’s a good example of something that’s both random and completely out of a team’s control. So let’s pretend for the sake of my argument that we’re back when whitescreen didn’t get a replay.

No amount of skill or effort can overcome a whitescreen in a BO1 bracket. If you get it, you lose (unless your partner can win in a 2v1 match which if so…, the level of competition in your region is not good)

However, the logical outcome of that statement is not that winners and losers are determined randomly. Rather, the likelihood is that the better team will win, but there is a chance that will not be the case. There is a problem in the system (white screens can cause the better team to lose through no fault of their own) which no amount of hard work can overcome. The only solution is a change to the competition system (e.g replays for white screens) to help decrease the likelihood of chance determining the winner rather than skill.

You would, of course, still expect to see good teams do consistently better than less good teams. You don’t whitescreen in every match, the refs don’t miss a call in every match. But when it happens, it is something beyond the team’s control which is determining the outcome of the match.

Now, with things like possible rules violations, it becomes even hazier. If you’re substantially better than your opponent, you can play safer to avoid potentially looking like you’re breaking the rules. So that helps tip the balance in favor of skill more. It’s a whole lot less clear than “my robot died and it became a 2v1 match”. But the same premise applies, it’s introducing a factor outside of the team’s control that might decide the outcome of the match.

Yes, teams should focus on what they can control, that’s the only thing they can actually do. But if there’s a problem with the competition system (issues with enforcement of rules) then the people who can control it should (within reason) do what they can to make the competition more reflective of skill, because it is within their control.

I’m receptive to arguments that this might increase the time it takes to run competitions, or create more problems than it solves, I don’t necessarily agree, but I’m receptive to them. I’m less receptive to arguments that amount to “it doesn’t matter if there’s a flaw in the competition formats, teams should just git gud so they win anyway”.

125

sadly, this still is. Not exactly on topic here, but there is a flaw in the manual that states that for a whitescreen to be grounds for a replay, the screen had to be white and all devices had to be flashing red, which simply isn’t what happens when a robot whitescreens.

This has been addressed in a q&a: G20 VI White Screen Devices Don't Blink Red : Robot Events but the rule was enforced as written for many events this season, including Kalahari. Many teams there lost matches, mine included, because the robots whitescreened and the manual technically did not permit a replay because only the screen was white, and the lights weren’t blinking.

I agree that whitescreens are a comparable problem to missed referee calls in a few ways, one of which is that there are multiple solutions/mitigations to.

ideally, whitescreens wouldn’t even occur because v5 brains would have been designed in such a way that they couldn’t happen. Just as ideally, the games should be designed so that referees don’t have to make as many difficult rulings.

However, a preventative approach doesn’t really seem to be happening for either situations, so we have to look at ways to mitigate these problems when they happen. Introducing replays for whitescreens (which is apparently going to be fixed in the February manual update) is a good way to reduce the effects of that problem. Difficult refereeing has multiple mitigation solutions.

Video replays are not something every event can offer, and I think we can all agree that it should not be something events have to provide. But if an event wants to provide video replays, I think it should be up to the event partner and volunteers running the event if they want to do so. If they have the ability and desire to take this measure in their event, it doesn’t seem productive to stop them. There are other potential mitigations for difficult referee calls though. Maybe if a referee has to make a call which is match effecting (as in, the outcome of the call would determine the outcome of the match), during an elimination match, they have the option to replay the match because they couldn’t fairly give the result to either team without being confident in the call. This would add time to events of course, but so does replaying matches due to whitescreens. And in the future, games will hopefully contain fewer difficult calls to make than this one, reducing the amount of time an event will add due to replays (perhaps only a few per event at most)

126

Thanks. This is the actual problem statement Referees are unable to always consistently enforce the game rules while due to lack of resources to support them being able to. is a potential diagnosis.

Referees across competition formats are unable to consistently enforce rules - we see it in @Jon_Jack 's anecdote about his hockey league, we see it in professional sports, we see it in youth sports. Humans are not consistent. Looking at your potential diagnosis, due to lack of resources we see, again, that referees, whether paid or not, with VR or not are unable to consistently enforce rules. Do referees with more resources (e.g. paid, with VR, etc.) more consistently enforce rules than those without? Reasonable arguments can be made that they don’t - I remember grousing about soccer referees when I played youth (where they were unpaid) and high school (where they were paid), and profession soccer’s VAR isn’t great currently.

As someone who has watched over 2000 Tipping Point matches (and I have the receipts) on YouTube, and several hundred in person (as a ref and spectator), I have not seen significant amounts of blown calls. I’ve seen refs who don’t know the game (literally were trained that morning), refs who think they know the rules (but don’t), but my observations would indicate that these are not terribly widespread.

At the worst-refed tournament I’ve seen this year (someone thrown into the job due to unforeseen circumstances who literally had to read the game manual during the tournament), it was still the case that the 2 most capable robots at the tournament faced off in the finals.

Have my teams been on the receiving end of bad calls this year? Yep - and they feel aggrieved by them. Have they been on the beneficial end of bad calls this year? Not that they can remember (but surely they were). Funny how human memory can work…

But going to more events minimizes that. I’d rather the RECF/GDC spend time and effort helping EPs put on more tournaments than think through and come up with VR policies. More tournaments, especially from EPs hosting multiple, allows the volunteer staff to gain experience. A volunteer reffing one tournament is probably more likely to miss calls (or not know the rules) than a ref who has done multiple events.

Having run the data, I am not at all convinced that chance (whitescreens, blown calls, bad schedules) plays a significant role in the outcome of matches or tournaments. Clearly it does happen some times, and that sucks. I believe @AlecMiller said earlier that he’d want VR for 3-4% of matches. At a tournament with 60 matches (say 32 teams), that would be 2-3 matches, resulting in maybe 1-2 changed results benefitting 2-4 teams (and “penalizing” 2-4 teams). I’d rather put resources towards helping that EP put on a second tournament giving each of those 32 teams (or give those 32 teams that were on the waitlist) another 60 matches. I think that would be more fun to competitors than changing 1 or 2 match outcomes.

Though I still believe it is a solution in search of a problem, I’d be open to @lacsap 's suggestion that those in favor of this approach the GDC about running a pilot and publish the results. I’m not sure what the “measurable” would be, since I believe this is an issue with what competitors perceive. Perhaps an exit interview/survey asking teams how many “blown calls” they think there were or how “fair” the tournament’s outcome was.

127

Measurable is Qualifying Match schedule using a match cycle that favors more matches per team (you know, so luck is not a factor in ranking because you had bad luck to have an randomly selected alliance partner that thwarted your prospects to Worlds… So if you have control group with skilled referee team and another group running VR at a drop of the hat - venue closes, does VR result in fewer matches per team, longer qualifying matches cycle or elimination rounds that EP paying overtime for venue use and grumpy volunteers being kept around longer than scheduled. Opinion aside, “measurable” includes number of QM per team, or longer average match cycle etc…

I do think interviews about impact for all stake holders is important.

One important point - this is NOT a GDC dependent test - any EP or organization that wants to run a formal experiment can do so under scrimmage format on robot events … that would get initial data… then you can proceed to have an official event use it in a blind test (dont alert teams that VR will be in play) see what happens…

this would help advance proposal vs wait until formal tests start - set up your own experiment.

128

Actually, by definition this sounds more like a random error than systematic error.

And there is no way to totally eliminate random error.
You can only reduce the effect by taking more readings and average.

In VRC context, thats where Bo3 (or even 5 or 7 or 9, etc) comes into play.

129

I wonder if this is then not necessarily a problem with needing video replay, but rather, better defined rules and better designed field elements?

For example for tipping point, the platform could have been designed differently so that there would be no way for a robot to go under it or affect it’s balance will void a lot of the complications with SG3.

Defining different terminology a bit more clearly, may help manage pinning trapping and hoarding.

Then if we are provided guidelines on what each of the 3 refs on a field should focus on, it would make it easier for a ref to make the right call. For example, when I was the head ref, I stood watching and following the robots in the neutral zone, the ref by the red zone focused on the platform there, while the ref by the blue zone focused on that side. Clear distribution of duties can help mitigate a lot of these issues.

The second thing would be the drivers meeting. When I led the drivers meeting, it was very much on not only clarifying rules, but also clarifying how our team of refs will watch the match, it allowed us to build rapport with the teams, being honest and saying it is possible that we make mistakes. So when we did, I am sure teams were frustrated, but they were understanding.

This would be even better if we can be provided the training course earlier so we can be more prepared. This way even if a person had to sub in as a ref, they could hopefully pick up the key points quickly.

I think video replay feels a bit more like a solution to an issue on the surface. A quick fix, a band-aid, to the problem at hand. There were definitely times I wished I could just watch a reply instead of spending 5 minutes on figuring out what happened, but often time these problems stem from players and refs not having a perfectly designed game, which I will admit is super hard given all the many possible what if scenarios.

There are many ways we can mitigate referee error, which is the core of the problem, and video replay can be one of the solutions if done right, however, there is more than can be done outside of video replay.

130

This is the core issue for me. It is an additional tool to make up for the challenge that is reffing, and it does NOT have to take additional time or require a full livestream. Simply having someone designated to record a match, who can pull up video to help me as a ref clarify something is all I want.

131

I agree it can be an additional tool. It is true for most good refs this is a boon, but the problem is when the ref is inexperienced, not as competent for whatever reason, or malicious and biased at worst. There are two reasons where it might take additional time:

  1. over reliance of the video replay by the ref, where they constantly defer to video replay for small things, or simply rely on the replay to do make judgements, allowing missed calls during the match, resulting in more rematches or annoyances.
  2. That one team that asks to check the replay every single time.

There are ways to design the rule to allow video replay so that head referees can watch them without disrupting time and enabling places to have different set ups of video replay based on the resources they have, while being consistent in the position of the replay. As well as helping mitigating abuse of the rule. for example:

It is optional for EP to have a live stream or video replay system. If the video replay can showcase the entire field including the players that is not blocked by players or objects, with a minimum resolution of 1080p, from a top down angle of between vertical downwards to 45 degrees, and can be retrieved within 1 minute, the Head Referee can decide to refer to the video replay to investigate rulings that may result in Disqualification or rematches immediately after the match and must inform the Event Partner. Only the Referees and Event Partner may view the video replay when making any ruling, players are not allowed to view or provide additional footage.

I am sure smarter people than I can determine the exact details and wording. The event partner was added to avoid abuse of video replays so players cant say the Head Ref is biased or whatever else, hopefully reducing malicious referees.

However, I do believe that a lot of the frustrations people are having, especially with the game this year, is clarity in game rules and better designed game elements that lead to refs making mistakes and missing calls. That there should be additional ways besides video replay to mitigate referee error.

At the same time, I must concede to the points made by Mentor_355v and Jon_Jack, that the points they make are valid. This won’t really change the final outcome of the match too much, making something “optional” isn’t really optional to some people, a video can also be biased based on who views it and from what angle, and the purpose of VEX is learning, not finding the best robot. I believe there is value in investigating video replays, but also to consider what is causing referees to make bad calls or miss calls. If we can reduce the number of things a referee can miss, than a video replay can be unnecessary.

132

Not quite sure how that would help, the EP does not have any say in calls during a match. Ref runs matches,EP runs tourney

Unless I got that wrong

133

There are several reasons I added the EP.

  1. It forces the head ref to pause and think if they even need to watch the replay, by creating a slight inconvenience.
  2. It makes it look much more official and can help teams see that the head ref is taking their calls seriously.
  3. It allows the EP to remind the head ref if they are behind schedule because of watching too many replays.
  4. it allows the EP to know how often and how infrequent the referee is using the replay, so they can monitor and see if there are potential issues with their refs, so they can provide the refs better support in future tournaments or change refs altogether.

The point is not that the EP has to watch the replays and make rulings, but to be allowed to if needed, because the referee is inexperienced or because teams frequently have concerns about the referees. It gives an avenue of oversight, just in case. However, as I said, I am sure smarter people can determine the exact details and wording. I am sure there are better ways to go about it.

134

No thank you.

EP’s attention is divided across many operational functionality of the event. EPs delegate responsibility of judging to Judge Advisor, officiating to Head Referee, Operations to TM crew, AV/Streaming to AV Production crew, division operation to Division lead, and volunteers to Volunteer Coordinator. oh yes, we also a liaison to facilities, security, etc. So please, understand, while EPs are there to make events possible, we put in place the best people to make the tournament experience possible. We step in when our volunteers can’t make it, and a whole slew of things. Like making sure there is food for teams at the event … the list goes on…

So forgive me. - EP is not the person you want in the Referee Video Review process - they might be dealing with a Code of Conduct problem, or backed up toilets…

This is why roping in EP into Video Review is not likely what you want if you want events to run smoothly.

135

Yes. I can count on the fingers of one hand on the number of matches I watch at any event. Their is so much to do and so many things pop up. I’ll pass on being part of the vid … sorry, got to go there is some issue with the food delivery.