Tournament managers: how do you run your judging of awards?

Our judging advisor is only there to instruct the judges, to get the started in the process (training), and help make a breakthrough if the judging process is coming to a standstill. Our advisor doesn’t do any judging or selecting of teams for awards.

Because the actual scoring numbers can vary widely from judge to judge, once we determine the top contenders, the design award rubric total score becomes somewhat irrelevant. Almost like the Excellence calculator… “Judges will use their best judgment” to select the Design Award winner.

Below are excerpts from the local judges guide:
Top contenders for Excellence will be found using the Excellence calculator. Once this calculation is completed, Judges will use their best judgment to choose the team they feel best exemplifies the best overall robotics program.

During the event, Judges should:

  • Review the Engineering Notebooks that were submitted by the teams. (Use the Design Award rubric)

  • Conduct student interviews and discussions with teams. (Usually done in the Team Pit Area)

  • Rank each team you have met with for awards consideration after meeting each team. Typically, rankings for the top 25% of the teams that you visited are needed during the deliberation process.

During judging deliberations, Judges should:

  • Post or share your top-ranked teams for each award, as instructed by the Judge Advisor. Typically, each judge team will initially post the top five teams for each award or one quarter of the judged teams, whichever is greater.

Design and Excellence have different criteria.

The Design Award is presented to a team that demonstrates an organized and professional approach to the design process, project and time management and team organization. Only teams that submit Engineering Notebooks are eligible for the Design Award. Key criteria:

  • Engineering Notebook is a clear, complete document of the team’s design process
  • Team is able to explain their design and strategy throughout the season
  • Team demonstrates personnel, time and resource management throughout the season
  • Teamwork, interview quality, and team professionalism

The Excellence Award is presented to a team that exemplifies overall excellence in building a high quality VEX robotics program. This team is a strong contender in numerous award categories. Excellence winners must have an engineering notebook. Key criteria:

  • Design Award ranking
  • Tournament Qualification Matches ranking
  • Robot Skills Challenge ranking
  • Other Judged Award rankings
  • High quality VEX robotics program
    A team does not have to win the competition to receive the Excellence Award, but must at least be competitive in the Judge’s rankings.

It is a “judged” award, there is always going to be a certain amount of subjectivity. Ther is NO way to completely remove it.

I think you can ease that credibility “problem”, when the award is presented to carefully point out what this award signifies. See AdamT’s response above. The Design Award in not for the best Robot, it is for the team that had the best design process and was able to best documentation of their process . It is entirely possible for a push-bot to win the Design Award.

I’m hoping I can add to this thread, without repeating too much of what has already been said. I’ve volunteered as a judge at Worlds twice and like to think I have a fairly good idea of what’s happening. The format they use at Worlds might be different to other competitions just due to the overwhelming number of high quality teams, but I think the process should scale down well. I should also add that I’m no expert and I’m only commenting on what I’ve observed.

First of all of like to say if you get the opportunity to judge at Worlds, take it. It’s a deeply informative and rewarding experience.

While teams are checking in, design notebooks should be handed in. Before the competition starts is the only time it’s quiet enough to go through them and judge them. Some competitions also set a deadline to hand in notebooks by, (up to a week before) this just allows judges to have a proper look through the notebook before design interviews.

Design Notebooks are judged from the design award rubric. (Which has already been linked in this thread). The rubric was rewritten recently to try make judging notebooks less subjective, but, of course, it can never be written out completely. You tend to find that opinions will normalise as you do more and more notebooks. Generally you have a set of judges (two judges) working on a notebook at the same time. All the judges can do this, not just those doing design interviews. Having judging sets also helps standardise the rubric scores. As notebooks are scored they are organised into two piles. Those above a certain threshold, and those below. Those above the threshold will be given design interviews, those aren’t didn’t make the cut. This is purely because of the time limitation and because design award is looking for the best notebook/interview. Design judges at Worlds now go to pits to do the design interview. With less teams to interview it means they have the time to go over each notebook in more detail after the initial judging process.

There are also Excellence judges and pit judges. Excellence judges at Worlds have an interview room and allocated slots for teams to be at the interview. Excellence doesn’t only take the interview into account, there is an excellence calculator on the REC website which outlines all the aspects that are taken into account. It is still a judged award so the top team on the scoresheet may not be deemed by the judges as excellence worthy as another team. But it is of course up to the judges to decide.

Pit judges (what I’ve always done) are the ones who walk the pits and ask questions for Build, Create, Amaze, Innovate and Judges awards. There are no set questions for these interviews and each set of judges is judging all 5 awards at once. The scoresheet has a small area to give team marks in different areas for the awards, but it’s best to write in details and take a photo as a reminder. As judging sets do more interviews they tend to get a good idea of which teams are better than others in certain aspects. Each judging set can nominate teams for any of the awards. At Worlds this is done by writing their team name on a post it note and sticking it under a sign for that award. Obviously they only nominate the best teams they’ve judged. In terms of identifying which teams have already been judged, a simple sticker on the pit sign is used. When all teams have been seen once, judges should reconvene to discuss the nominated teams (probably about 6 for each award). Each judging pair gives the reason they nominated that team and those nominated teams are seen again by a different judge set. Judges reconvene again and teams are ranked as they are discounted for the award.

The reason teams are ranked is in the chance a team becomes ineligible. This could happen for a variety of reasons. Also being well judged by pit judges helps teams towards excellence.

Hopefully that covers the process. Tarek has released a whole load of useful information which is on the REC website under EP documents and then judging documents.

Shout out if you have any questions.

I think the photo is a great idea. I’m amazed sometimes at the number of judges who take very few notes, so a photo on an iPhone can go a long way toward eliminating some confusion about which team did what.

Also, generally speaking, thanks for your insights on this.

I myself am not a great note taker. I like to listen and try get information out of teams and not distract them in the process. Then at the end I’ll thank the team for their time, give them a sticker and ask for a photo of the robot.

Once out of earshot I’ll discuss with my judging partner and write a few notes (especially for the standout teams) and put in numbers in the judging boxes. That way the conversation is fresh in your mind.

The photo stands as a reminder of why you’ve recommended a certain team. For me, a comment of ‘cool chassis’ with a star next to innovate award might be my only clue as to why I’ve recommended them. Seeing the robot again is what jogs my memory to the conversation.

Everyone is different, I know this works for me. It probably won’t work for most people.