Replays/Video Review

#1

As we all know, judges are volunteers that put aside their personal lives for full days to help out at tournaments. However, this is accompanied by some degree of imperfection.

Over the last few days, I’ve been fortunate enough to compete at the Create Us Open tournament and get to play with and against some of the best robots in the world. (And Pigpen doubled up their chain so that was pretty cool too.) But because judges are humans who make mistakes, I want to suggest a system for video replays. Before I jump in, I’d like to point out a few examples of irrefutably incorrect judging (not my opinion on close calls- just objective facts) to highlight the need for this kind of evidence. My intention is not to undermine or humiliate specific referees, but rather to show overall how hard these calls are, both for the refs and for the students.

-Orange QF 1-1, the refs ruled auton a tie when really the red alliance had more points. The blue alliance won the match by 1 point with auton counted as a tie.
-Green SF 1-1, the head ref began multiple pin counts when robots were pushing each other around the field with no confinement of anyone to one square. They ultimately DQed the blue alliance when there was objectively no pin that lasted for more than 5 seconds.
-Red QF 2-1, the refs made no entanglement call when rubber bands on a blue robot’s intake entangled with their opponent’s chassis for more than half the match, as the blue robot was playing defense against the red robot. Red controlled all 6 high flags right before the entanglement, but ended up losing the match by 2 points.

And I’m sure there were other examples of poor reffing, but these were the 3 that caught my eye the most.

In elims, each alliance can call a 3 minute timeout. Rather than their time out, what if the alliance was permitted one 3 minute grace period to present video evidence to the refs? The footage could come from a cell phone recording of the match or from official lived streamed footage, and it would only take a max of 3 minutes.

If you think 3 minutes is too long, think about the amount of time the students put into their robots before the competition. Wouldn’t you be upset if a robot you had put 1000 hours into was DQed unfairly or lost because the refs miscounted auto? 3 minutes is very little time compared to 1000 hours.

What does everyone think?

15 Likes

One more reason to love BO3
One more reason to love BO3
#2

I like a lot of things about this idea (less stress on teams, and less stress on referees in some cases), but it would likely give an advantage to teams who are able to get better match footage than other teams (such as by having more people to take videos, or by having access to better cameras).

It would work better at an event with a livestream that is easy to rewind and replay (which may not be the case for most livestreams), at least if it’s a one-of-a-kind event like Worlds or the CREATE U.S. Open (something where one event could use this system without other events needing to do so).

7 Likes

#3

I think allowing refs to review footage would be a major improvement. Every once in a while, a ref makes a mistake and this is normal. Reviewing footage can help correct these. Also, in a case where the ref is deciding on a possible penalty, the ref can view the footage to refresh their memory and in turn make the best decision.

7 Likes

#4

By the way, I’d like to mention that based on the original post, this thread seems like it’s mainly about the idea of letting teams use a 3-minute time-out to show video evidence to refs, and not so much about allowing video replays in other cases (which has been debated in other threads).

There are many different ways to allow video evidence in some cases while prohibiting it in others, like limiting how much time can be spent looking at a video, where the video can come from, or whice matches video can be reviewed for. Each of these options has benefits and drawbacks.

Most of the proposals I remember hearing in the past had drawbacks that have prevented them from being implemented. The same might be true for this one, but I believe it is worth it to keep trying, as long as we keep the conversation civil.

Keep in mind that referees, EP’s, and other volunteers, along with students, parents, and coaches, at all levels of experience and funding, each have needs and desires that those in other positions (or even other regions) may not know about.

Also, many EP’s and referees also participated in VRC as students (or coaches), and many current VRC and VEX U students (and even more coaches) have also been referees at VRC events, so someone on one side of the debate may know how it feels to be in the other side’s shoes, even if they disagree on the best way to balance both sides’ wishes.

Edited to add: Also, even if someone hasn’t been in your shoes themselves, they may know someone who has, or they may be able to figure out how you feel based on other things they know.

2 Likes

#5

I somewhat agree, after all refs aren’t perfect and they could easily miss some important detail, especially in a game as chaotic as this, even if they’re trying their best. But then again, change is not likely to occur. I feel that teams rarely use their time out, but if it was video review time instead, teams would probably use it whenever possible. And lots of scenarios, even on video, could force a ref into making an extremely hard decision where no solution pleases everyone. Maybe only allow video evidence for eliminations? This would save a lot of time and effort, as well as strengthening the weaker parts of the system.

3 Likes

#6

At the end of the match the referee goes “Auton is a tie”. That was the time for the red alliance to go “Hey wait, please recount”. I kind of doubt the ref said “tie,driver control,3,2,1 go” in a second that it would take the red alliance to count and start a “Hey, wait, recount please” moment

Pin counts are easy to fix, back away. Just back away. Score some place else for 5 seconds.

And who would you have called entanglement on? Red since they were clearly entangled with the blue robot? Or blue since the red robot was clearly entangled with them? Was there something keeping the red robot from driving away and breaking the rubber bands?

All good questions. I’m personally looking forward to next year’s Create US Open where @Anomaly will be a referee and help out with these calls.

Video replay has been talked to death (although not as much as @B03 has been talked to death (*) ) and it’s hard to setup. hard to run, it’s hard to manage, and the rules around when to do it is hard. No idea on how many fields there were at the Create Open, but adding cameras to each one is a huge pain. Use of someone’s shaky, janky screen shot of their phone is not the place to start.

But, heck, I’m a huge fan of pilots. Ask nicely to see if you can offer a video pilot where you’ll record all the matches. And IF the REF wants a replay they can come look. Who knows, maybe the ref will.

But again as a EP that runs a ton of events that has limited volunteer help, this is a hard no for me.

( edited to add: And it appears “The Discord” is all hopped up with comments about a lack of referees. So if I had a chance to put people on recorders, I’d pick putting Mark I eyeballs on the fields to watch the direct action. But again @Anomaly this is an opening for you to help out)

(* Wasn’t really offering up that we should proceed to beat the no video rule as much as the B03 rule just in case you were wondering :roll_eyes:)

8 Likes

#7

So this is actually an interesting concept.

I think the hard thing however really is a lot of the logistical hurdles it takes to achieve this.

One of the strengths and also weaknesses of this program is the great variance you see in the level of events.

For the newer EP these sorts of things are hard to implement, particularly for more resource-starved EPs and programs. This isn’t feasible.

For those experienced EPs this is still a challenge and expense. ’

It’s a challenge for multi-million and multi-billion dollar industries.

You also need standardization across the board, and certification really for people who you call on to even review video.

It’s not only a lot on the EP, but a lot for the RECF to oversee and organize.

Even with their almost 2 decades plus they have on RECF, FIRST still doesn’t have that done, and the technical portion of everything there is at a higher value.

I was at the CREATE US Open, and I’ve refereed, and been a head referee at multiple events, including mega events, but this was honestly one of the more disappointing experiences I had.

I know even in the Open Division, rules were misinterpreted if not downright incorrect, or made up, even in a modified division.

There was a pin called in the HS Division that wasn’t a pin. And I didn’t see a real confirmation of the count towards the teams.

I remember last year there was an unwritten but enforced rule on defensive autonomous, when that was not communicated to my teams, and when my teams had questions were responded to in a hostile and antagonistic manner.

So I’ve seen 2 straight years of inconsistent and incorrect calls at an event I love coming to.

Back on topic, if you want greater transparency or effectiveness of refereeing calls, there’s two ways of doing it.

Ensure you either have an RECF staff member to ensure those calls are consistent, or have paid referees certified by a board.

That financial expense is a blocker to implementing this.

5 Likes

#8
VEX Robotics Competition Turning Point is an interactive game. Some incidental tipping, Entanglement, and damage may occur as a part of normal gameplay without violation. It will be up to the head referee’s discretion whether the interaction was incidental or intentional.

I know you are more concerned about the replay system rather than the specific incidents but unless the blue robot was intentionally entangling the red robot, which is hard to prove but unlikely, the interaction would be legal. So don’t say “irrefutably incorrect judging”. I didn’t see the other two, so I won’t argue about those.

I do understand your points and your ideas. I have reffed in the past and I personally wouldn’t mind looking at footage to review a match during eliminations when there is a disagreement but unless live footage is standardized at all events I think that it would not be a good idea to let teams film their own matches and bring that as proof.

4 Likes

#9

That’s too bad, sorry to hear that!

2 Likes

#10

I’d personally be a fan of replays. There will definitely need to be restrictions if video review is implemented. If reviews are allowed for unlimited matches whenever teams want, it will get out of hand.

Maybe restrictions along the lines of

  • only match affecting
  • possibly only during eliminations?
  • only x number allowed during a tournament
  • must have specific infringement of the rules for refs to review
4 Likes

#11

It seems to me like the problem is not these specific cases, but that having had video review as an option in these could have helped. (And video review would really only have been helpful in the first two, the last case is a bit iffy.)

In the case of the Orange QF 1-1, I have seen cases in which the situation you described can happen (for example at Google). In Q13, auton was a tie, however red was awarded the bonus directly after the auton ended, going directly into driver control. It took me about 15 seconds to see that it was a tie, which the alliances were definitely not given.


This is a thing that happens (in this case the auton didn’t affect the match, but I used this match to exemplify just how fast refs/announcers score and move on).

In the case of Green SF 1-1, pushing (not pinning) can definitely affect the outcome of a match, and teams should never be penalized for pushing. Pinning is different, if a team is pinning, then of course they should back away to do something else. But teams have the right to push. In this case, video review really could have helped.

In Red QF 2-2, I agree with you. Unless the entanglement was on purpose (which is borderline impossible to prove), it was incidental gameplay, and video review probably wouldn’t have done much.

My points here were not intended to argue in favor or against specific teams, but to highlight examples where video review would have helped.

4 Likes

#12

Even the World Cup showed the future of soccer with VAR (video assistant referee) in 2018. VEX competition should change with the times…

2 Likes

#13

Although this thread is about replays / video review, the reason that this is being proposed is to correct referee “mistakes”. As I explained on other threads recently, the REC Foundation is actively working on a referee certification program. To start, we will require all referees that ref at a VEX Worlds qualifying event to earn this certification. And the REC Foundation will work with EPs to send a certified Referee to Worlds Qualifying events where they need assistance. Additionally, all referees at the 2019 VEX Robotics World Championship have worked at least one Worlds qualifying event - that is a change we made last year. I really want to evaluate these improvements first before we start working to implement a video review program. However I will add this as an agenda item for the EP Summit.

On a side note, I received a lot of texts, emails and PM’s yesterday about referee calls at the CREATE US Open. The US Open is not associated with the REC Foundation, but I truly appreciate CREATE hosting this event. Both the RECF and VEX Robotics do send staff to provide technical support. I myself was there for two days. I know that CREATE works extremely hard to put on this event and I am sure they will evaluate all the feedback that they receive. I met many of the volunteers that support the great CREATE programs in Nebraska and Iowa. They are a fine group of people truly dedicated to providing an opportunity for students to compete in robotics with teams from across the country and China.

19 Likes

#14

The World Cup is run by paid professionals, with a massive AV budget that dwarfs even the AV budget for VEX Worlds.

On the other hand, most VRC tournaments are run entirely by volunteers, with little to no budget for fancy equipment.

7 Likes

#15

Every time video review comes up people reference professional sports. Why? A more accurate comparison IMO would be high school football, basketball, soccer, etc. For the most part high school sports do not do video replay, for the same reason VEX doesn’t: cost, complexity, etc. I did find this though:

However, football games are put on by schools and have a relatively large budget (compared to most VRC events). A lot of high school games already get broadcast on TV thus they have multiple cameras and angles and such making replay feasible. Most VRC events are not broadcast at all, and those that are tend to be from a fixed camera, some with pretty questionable quality.

Cell phone video from spectators is really a non-starter.

To be clear: I don’t think anyone who says replay isn’t a good idea is suggesting that wrong calls aren’t made or that replay couldn’t perhaps overturn those wrong calls. The reason @Foster and others say replay isn’t really an option is simply the logistics: time, equipment, training, etc. Not to mention that it will inevitably lead to even more arguing with event officials. How many times do people watching pro sports see the refs overturn a call on replay and people still get mad and say the refs blew it, are “blind”, etc? Pretty often in my experience… (in other words, video replay isn’t magically going to make contention over ref calls go away, in a lot of cases it will just shift which alliance is upset).

10 Likes

#16

I pretty much agree with all your points. It is impossible for 90% of events to do video replay without relying on sketchy cell phone footage. The current rules however just say a ref isn’t even allowed to look if he wants to and that is a bit strong. I have reffed events, and I knew where I could get footage, but was unable to look.

If I was going to propose a new system it would have some big rules.

  1. Events have to opt to allowing footage review. (its no by default) All teams must be told during the drivers meeting that a tournament has opted in.
  2. The only cameras to be used are those officially belonging to the event. Unless the head referee specifically requests to look at other footage.
  3. Teams may request a ref reviews footage after a call. However he does not have to listen to your request. (similar to the request to resize an opponent) It is left up to the head referees discretion.
9 Likes

#17

I think that’s to protect you (as a ref). If it’s your choice, then you’ll have teams bugging you after every match to look at the video, and it’ll be yet another judgement call for you to worry about and teams to question you on if you made the right call (i.e. whether to look at the video or not). When it’s a blanket policy, it’s out of your hands and you can just say “Sorry I’m not allowed.”

I think the other aspect is a bit of fairness. If some events have it and others don’t, that potentially makes the events that have it “better”. That, in turn, puts additional pressure on EPs (“Event XYZ has video replay, why can’t you do it?”). Again, I think this is to protect everyone, so cash-strapped events don’t feel even more pressure to spend money that they really don’t have providing this service.

Two years ago at our event we were actually doing instant replay for the audience and webcast (we didn’t have enough time in between qual matches to actually show it, but we did use it during elims). A team came up to me rather upset because they were convinced the ref had scored auto wrong. We talked to the head ref who basically said “Sorry, it was a few matches ago. I’m pretty sure it was right but I’d have no way of knowing for sure” thus the call had to stand. Out of curiosity more than anything, I pulled up the instant replay footage from the match (I was not a ref, and would not have shown the ref regardless). My intention was to either confirm it was right or else to notify the ref - not to change the score because that’s not allowed - but just to make sure he’s double-checking things. Luckily he got it right, and in that case I brought up the team to show them.

What I learned from that was that it was surprisingly time-consuming to do. Even though I had captures of each match and knew which file it was, it took a few minutes to open the file and scrub around to the right point (I think the question was regarding autonomous scoring). I think if replay was available at an event it could easily be a full-time job, perhaps even more than 1 volunteer depending on the size of the event. It can be tough to find volunteers as it is, and I don’t think this is something the refs themselves could do unless you added extra refs to make up for it.

8 Likes

#18

I’m just as imperfect as the next guy. Video replays would make it easier on refs as well because there wouldn’t be so much pressure on them to get it right. I’d rest a lot easier if I knew my calls were correct.

Why is it hard to review someone’s phone? What does it even mean for a video to be “janky”? lol

This reflects an unsettling disconnect between you, an older man who never competed in vex, and your refs and competitors. If this is really how you feel, I urge you to talk to some of your volunteers and hear how their thoughts about video replays.

Honestly I’m not sure why cell phone recordings are so objectionable. It’s not like you can convincingly fabricate match details on an iphone in 2 minutes. That would take a CGI team and a lot of hours. And as for certification, what about the refs themselves?

I don’t want to get too caught up in the examples, but rubber band intakes without mesh or some other form of protection have been clarified to pose an unnecessary risk of entanglement on the Q and A, and the offensive robot gets the benefit of the doubt. In the case where a defensive robot entangles the offensive opponent with a mechanism on the defensive bot that poses a risk of entanglement, and the entanglement is match affecting, it’s a textbook DQ.

Thanks for the perspective!

That seems to be a common theme with a lot of posters. So let’s start with events that are streamed and we can go from there. How does a replay system at Vex Worlds sound?

This seems totally reasonable. My suggestion was only once per alliance, only in eliminations, and with a time limit. We could fine tune the specifics, but certainly there need to be some reasonable restrictions so teams can’t request a replay every match.

Dan, thank you again for staying transparent and being open to feedback and criticism. Your post sheds a lot of light on the changes the RECF is making to better the quality of reffing at worlds.

At the same time, we both know that reffing is an incredibly complex and multifaceted job. I doubt anyone who has reffed at multiple events can say with total certainty they’ve never made a wrong call or missed seeing an important detail. In my opinion, video replays would help to compliment, not to undermine, the expertise of the volunteers that devote their time and skill to make the events run as well as they do.

As for Create Us Open / Nationals, I’m just going to openly say the reffing was abysmal. Seeing how badly reffing can be screwed up makes me appreciate more than ever how good a lot of refs are and how tricky their job is. I think a lot of us would agree that video replays would relieve some of this stress, both on the competitors and on the refs, and make the event run more smoothly for everyone.

Again, thank you for the post.

Cell phones are fancy equipment that the vast majority of EPs, refs, and competitors already have :slight_smile:

These are good suggestions, thanks for the perspective! If I can ask though, why is cell phone footage an issue? Isn’t it showing the same stuff?

4 Likes

#19

???
I’m not sure where you saw that, but the Q&A on this subject ruled mostly the opposite:

These rules all combine to form the following overarching guideline regarding “rubber band intakes”:

  • A Robot with this type of mechanism is assuming the potential risk of Entanglement.
  • Teams who build mechanisms with Entanglement hazards are responsible for minimizing this risk, or accepting the potential inevitability of becoming Entangled with an opponent.
  • A Robot with a “rubber band intake” who becomes Entangled with an opponent generally would not result in a <G12> violation on their opponent, because it is inherently the rubber band Robot’s “fault” for assuming that risk, per <G12>
  • However, that “fault” does not immediately flip to a <G12> violation on the rubber band Robot, because this intake is generally being used for an inherently offensive maneuver, per <G13>
  • In short, under normal gameplay (as judged by the Head Referee), there would be no violations on either Team.
  • Of course, all of the above is superceded by <G12> if the Head Referee determines that the Entanglement was intentionally or egregiously initiated by either Robot (e.g. the rubber band Robot is defending and “intakes” their opponent with no Balls nearby or other reason to have the intake; or, the non-rubber-band Robot is defending and puts out a “claw” which is immediately ensnared by the intake).

https://www.robotevents.com/VRC/2018-2019/QA/262

1 Like

#20

Wrong side of the line. Way over the wrong side of the line.

Even for you @Anomaly this is way over the top.

Let me clue you into a few things that you should think about

  1. Yes, I’m older. I’m an old. That means I have a boatload of real world experience. Day after Day, Month after Month, Year after Year of learning stuff. Since I was a cranky teen until today as a cranky old. After raising kids, the common thread is “Wow Dad, this adult stuff is hard”. Looking forward to the day you tell your parents that.

  2. I’ve been doing this for 13 years, so I was a Middle when I started this. I started to help my child and got hooked on it. As an engineer it was an easy thing to do. Since bolting things is something that I did for years, it was an easy transition.

  3. Competition robotics is worthless if there isn’t competitions. So another mentor and I set up a non-profit to run competitions. We started off with 2 and across the years grew to 12 a year. And the years past so I’ve been responsible for over 100 different events. Cool thing is that I learned year after year as I got older was how to do events bigger and better.

  4. Volunteers cover a wide range of activities

  • Planning
  • Finding 50+ volunteers
  • Advertising
  • Getting shirts and hats for the teams
  • Pit setup
    ** tables
    ** coverings
    ** power
    ** signs
  • Food service for setup night (Food and water)
  • Borrowing fields and setting up
  • Electronics setting up
  • Juice, Water and breakfast foods
  • Checkin and inspection
  • Team queuing
  • Judges
    ** Head judge
    ** Judge training
    ** Foms and documents they need
  • Referees
    ** Shirts
    ** Training
    ** Connecting to make sure they are making good calls.
  • Event continuity
  • Announcing
  • Scorekeeping
    ** Form management
    ** Making sure that scores are recorded correctly
  • Computer management
  • Awards
  • VIP management
  • Sponsor management – Did they get their stuff are they happy
  • Daytime food
  • Daycare management for younger robot families
  • Afternoon food for volunteers
  • Wrangle the press for awards and match videos
  • Start teardown
  • Get all the people, equipement, fields, power, chairs, computer stuff packed up and sent away
  • Close out the venue with the management

Just off the top of my head this is the people that are working.

Oh yea, run the event. So based on all the above, about 8% of the people are referees. I’m sure video replay not on their minds. It’s not a blip on mine. Only a blip on yours.

I spend about 100 hours my time making an event work. Think about that @Anomaly, lets drop two and a half weeks on paychecks on helping other people.

So I don’t appreciate the remarks. You are clueless. You have written to me and other EP’s about how you talk to us and we get apologizes. Hours later you are back calling us out for being stupid.

You need to pull your act together. Your friends on “The Discord” need to pull it together.

Lots of the EP’s are here trying to make events possible, make the best possible events.

Except I have you here standing on the forum making my sneakers wet? What’s up with that?

@Anomaly I’m not really up for yet another insincere apology. Maybe walking away until after Worlds would work?

And I’m not going into all the hours that I’ve spent across the years developing teams. Lots of teams to worlds. Lots of teams that win awards. Lots that win Excellence. Two of them win Mentor of the year. Lots of inspire to lots of roboteers. College grads for roboteers that were not going to school. A detention girl that pulled out and is now a nurse with a family. Little successes that mean nothing to you, but mean a lot to the roboteers.

And just a reminder, getting old is a good thing. Not getting old is called death. I learn something new every week, my talents expand. So for me, not getting old will suck, since there is so much to learn

And I’m sure Sharwin from 5776E, your comments will add to this conversation.

24 Likes