At our last competition like 6 out of 11 awards were handed to teams from the host organization. And like 3 of the teams at that competition ended up going to worlds. I didn’t read much from the comments above, but at states, our sister team’s judge during interviews was the spouse of the head ref. And the design award was handed to guess… Yep a team from that host organization from earlier.
I was involved in a questionable decision to some, but I talked to state and REC post decision and explained what happened in the decision making process in more detail. In the end it was determined the judges did their job correctly and all. Conflicts can be dealt with, just some teams cannot accept that the judges might not make the decision you want or think should be made. There are many times I disagree with the decision and the reason the judges make those decisions. My job as a JA is not to agree with the decision, but to be able to defend that decision so I need the judges to get me to that point regardless. Ideally we would have judges from industries to help, but you will never totally escape the bias perception no matter what you do. The judges are free to decide how they want as long as the boxes for the award are checked.
Case in point. Innovation is a lesser award than Excellence and Design (not that I agree with it). Some judges due to the uniqueness of an Innovate refuse to allow them to be considered for the top awards. It is a rough time.
Add in the EP’s generally think the decision process takes 5 to 10 minutes, a JA has other issues to juggle while trying to lead the judges in considering things.
I want to have a JA only thread and at the state level meetings to discuss things to get it more standardized. As always I want to hear positives and negatives so I can improve things.
I commend your efforts in trying to improve things.
May I suggest a standardized submission method of notebooks? All notebooks should be submitted digitally to the EP. When the event ends, the EP will no longer have access, but the notebooks are kept in a repository owned by RECF for the rest of the season. This will leave a trail of notebook submissions from each team, making it more difficult to fabricate or back-edit notebooks. The trail of notebook submission history of each team should be visible to judges at all competitions each team attends. Especially at big events such as State championships, Signature, and World, for teams that are in consideration of top awards (or all awards), the judging process should include the judges looking at the repository and verifying that the notebook trail is consistent and always forward edited.
This of course would require some work and investment on RECF for data storage.
I forgot to mention that I also agree with you. I believe that Innovate should definitely be higher up, maybe even higher than Excellence; Innovate as the highest award is excellence with an innovative feature as well.
I’m a coach and ran an event as EP this year, I’ll try to keep my version of this shorter:
-
Judges are REALLY HARD to find. We reached out to every local college and had 0 volunteers, teachers were unable or unwilling to commit, and in the end, every judge we had was a parent or family member of one team or another. Our JA was a coach from the neighboring school district who also had kids competing, which helped balance that playing field, and I’d argue is the only reason I’d be willing to accept the conflicts.
-
Most journals are not well developed. This is particularly true at IQ level, but quite a few of my teams had poorly written journals that didn’t even meet the bare minimum expectations. I’ll typically have one out of 5 teams even both writing a journal that isn’t just a daily “we met and changed our intake” with no details style journal in my own district, and we routinely have teams at state.
-
The lack of feedback SUCKS as a rule. I get the idea is to separate judges from any direct feedback of a “bad call”, but at the SCLA tournament in St. Paul this year, they made the decision to not give out judged awards and give feedback instead. That feedback was amazing and really gave my kids the chance to improve their journals and their behaviors and see where they went wrong. As an example, one feedback item was that kids weren’t taking out their headphones during interviews. They paused their music, but the judges didn’t know that and felt disrespected.
-
The lack of a points system really incentivizes gaming the system. Our teams were some of the first to figure out that you could just run skills auto starting in the parking zone, spin your intake, and get 15 points. Half our teams got state invites from skills rolldown even though their bot couldn’t score. To be clear, this isn’t to put down those teams, they took full advantage of the rules to their benefit, but if you used a points-based system like FRC or FTC, then did rolldown based on that, one judged award or just a good skills strategy due to a weak game rule(excluding excellence) wouldn’t be enough to qualify if your performance is otherwise poor.
-
Judges do NOT understand the award meanings. At one event I was judging, the judges though the awards were tiered excellence → design → think, where excellence was 1st place, design was 2nd, and think was 3rd, and I had to explain the think award was for programming in particular and it caused a re-interview.
In summary, the current system is flawed due to a lack of highly experienced judges or those willing to put in the time to volunteer for it, and the loopholes in how the system works encourage teams to game these awards/skills rather than a genuine best-effort on it.
When there is conflict, there will always be doubt in the outcome. Even though RECF usually sides with their EP’s, it is worth voicing concerns just so they are aware that enough people notice. I’ve never had a concern validated by RECF, however when I do contact them, it is with the intent of reporting because some noise is better than silence.
With all the AI Innovate is the only award that truly will come from the students and should have the most impact over all in my opinion.
What does innovate award really look for? That’s the key differentiator. I witnessed that a team won the Innovate Award for one of their innovative ideas in the robot, which they no longer use. The innovative idea is not performing, and they were not even in the top 50% for skills or teamwork. If innovate is all about taking an innovative idea, they can get the idea from AI. IMHO, innovate should consider team performance on the day of the event. Thoughts?
Totally agree. And since Innovate already has a prerequisite of being a Design award contender, it really should recognized at the highest level. It is difficult for student-designed robots to make top 40% at higher level competitions, so robot performance should not be attached to Innovate. However, this difficulty is only due to having to compete with the meta-bots/clone bots.
I somewhat answered your question in my previous response
Most contenders for Innovate will have a hard time with top 40%, maybe 50% is possible. Right now, the requirement is that the Innovate feature must be in use and is a major factor in the game strategy of the team.
I will be honest. Based on the way I run the judging, the criteria are fully developed Innovate (In discussion for design not always top 2 or 3). UNIQUE!!! Something that can be or has been demonstrated at the tournament I am working. Having said that I also know notebooks are NOT graded consistently between events. This is something that irks me to no end. The rules state things and through personal, live conversations, I cannot get a straight answer for how long the notebooks are given to be scored. Last year, my times were no more than 10 minutes to prescreen. 30 minutes to grade. If the rules are not changed, it will remain the same. Innovations are reassessed after initial interviews and narrowed down to a very few. My judges do both the notebooks and the interviews so they have a solid feel for it. Interviews are given equal weight to scores so the top 40% rule for excellence does factor into decisions which can impact who gets the Innovation Award.
Absouletly not. My friends dad is the judge with 3 years of judging but as soon as his kid starts first ever comp. Suddenly, a judges award and bro had a huey but the claw wasn’t working
“Right now, the requirement is that the Innovate feature must be in use and is a major factor in the game strategy of the team.” – I have not seen this considered in the rubric or judging deliberation. I wish this was made more explicit.
I understand the challenges in getting volunteer judges. Everyone is busy and hardly has anytime to spend 6-8 hours.
My suggestion:
-
- If parents or related members are the only option, make sure you include a equal distribution of parents from all the local clubs or all participating clubs. This will ensure that every local club supports each other and we grow together as a community.
- if judged awards are included, cast a wider net to get judges. Reach out to high schools or universities and get students. Have tie up with professors where they can assign students as judges at least for State championships or Signature events
- Consider remote judging from judges across different state. May be centralize the judging and have a pool of judges who are willing to volunteer on certain day/ division.
It’s in the Judge Award Descriptions sheet. And if you read the difference between Design and Innovate, it is actually more difficult to earn the Innovate award because besides a well documented notebook, and good interview, it also require some unique feature/design element.
https://kb.roboticseducation.org/hc/en-us/article_attachments/34300691647383
While you are technically correct, the award ranking are Excellence, Design than Innovate. I have heard the Innovate is getting overhauled. I am hopeful the ranking requirement is removed for the notebook as the use of AI is rampant. The Innovate should be fully developed and explained. The bad part is if teams are submitting over 1000 slides and the JA actually applies a time limit to review notebooks things will get missed. Personally I list a brief description of the Innovates claimed. This is on a printout I give to my judges so they can look for them and compare to other robots. Some teams claim features several teams have. Others claim things that do not work. So I have a plan to improve things for my events for these, but the list did help sort out innovations versus teams claiming a common thing and application of it. There will always be room to improve. Consistancy from JA to JA would benefit everyone.
Yeah, exactly what I was thinking!
Why does the double qualification have to be awarded based on the driver skill and not the design process, the build, the documentation and the bot? I would like to see VEX bring about a change here, because the prospect of building and developing STEM skills within the programme is shifting towards an eerie end of teams rummaging for teamwork or skills trophies. If I have got anything wrong, can any of the referees or coaches please correct me, since I’m still just a year 8 student?
I like this a lot.
In fact, all initial interviews and initial notebooks could be judged ahead of time by people not affiliated with any local program. Ideally, there would be a program where every team submitted a volunteer for the program that had to do a set number of hours in the season. Then the volunteer would be used for events outside their region. All notebooks could be judged that way as well. Only second interviews would be done at the event. We do remote judging for our events as much as possible, but volunteer recruiting is pretty tough.
Several have mentioned AI.
I don’t love AI for lots of reasons, but I wonder if JUDGING could be made a bit more neutral by having AI search each notebook for specific parts that correspond to the rubrics, then present these to judge panels in anonymized form (perhaps extracted by rubric entry–so they see only the parts of the notebook with the pertinent info). Then AFTER those have been looked over, it could direct judges to the correct notebook to find the extracted text.
I think this could speed up judging and also make it a bit more neutral (after all, the AI doesn’t have any ‘skin in the game’).
This is just an idea–I am not saying it will solve everything, or is even the best approach (please don’t flame me!) Just a point to add to the discussion.
NOTE: I am NOT saying to turn all the judging over to the AI–just leverage it to assist and ensure notebooks are evaluated in a consistent fashion.
AI will hallucinate things. This makes it unreliable.