There are a few issues I have learned about when it comes to remote judging. One the time taken to review notebooks is substantially longer than in events judged at the time of the competition. Thus it makes it an unfair event. Next, the notebooks really lead to better questions in an interview setting. It helps weed out those who know from those who don’t. Body language also gives away things so…in person is better. Getting volunteers is horribly hard. Getting the few parents or local businesses to help is difficult to be kind. As for fairness…until the notebook length, notebook review time, and interviews (how they interviews are used in the debate) are corrected. There is not going to be a fair solution. If the guidance is 30 minutes per notebook (last year’s) explain to me in some detail how multiple over 1000 slide notebooks are accurately reviewed… I have been blasting the people responsible for the rules for 2 plus years now to fix this. Like I tell them. If the teams know they are capped at 500 slides, those slides for the top teams will be powerful. For the teams using these long notebooks for college applications, I would say that the competition of robotics is not the place nor should it be for this.
Because skills is the only thing within a team’s control. Robotics isn’t just about driving, but it isn’t just about judging either. Judging is already awarded before skills drop-down is, and removing skills drop-down would make qualifying for competitions mostly luck based. Judging is one of the most random things, because of different volunteer perspectives, one judge being more vocal about their team than another, and because they’re humans. If team had to solely rely on this system to qualify, they would be a lot more stressed and less certain to go to state or worlds. Skills is one of the best ways to test a team’s robot design (tournament testing driving, and judging testing design process) so I think it’s fair that it gets its share of qualifications. And tbh, most judged awards are already used, if your going to award judges or sportsmanship, I’m sorry but you can’t convince those teams are more advanced at robotics in general than 2nd place skills.
You do have to give your judges guidelines about how long to spend on each notebook. That is standard practice at any event I’ve been involved in remote notebook judging. Teams are definately going to be aware of the fact that if they have a thousand-page notebook that they will need some sort of tabs or sticky notes to guide the judges through it. No one would expect a judge to actually read through the whole thing.
I think this is appropriate in the second interview, but for the first interview I don’t agree that the judges should have to have reviewed those team’s notebooks first. The interview should stand on its own as the notebook does. We will see with the new judging materials.
I think it’s easier to get judges on a Tuesday evening from home is a lot easier than getting volunteers at my school on a Saturday morning.
Based on coversations I have had with JA’s that have remote notebooks…Times are conveyed but no method to ensure those instructions are followed.
On this I believe we will need to agree to disagree. 10 minute interviews 5 for the team and 5 for questions. When you are to screen for non student writing and all, the notebook’s language and structure need to match the interview.
Regardless of which day, after work volunteers that have kids or expertise desired…There is not going to be much of a change. Engineers, scientists, and most STEM jobs seldom have much time after working and troubleshooting.
From my experience, we judge the notebooks during set up of the event. We start interviewing post inspection and go. The table of contents, tabs or whatever are nice guides to get to certain points, but those are also more developed that other areas of the notebook. If the notebooks are too large, the grading just needs to end at the time given (recommended 40 minutes so far this year), If that means teams with large notebooks get 0 because time runs out, that is a risk that team takes. The review times should not be based on the size of the notebooks, but declare prior or match the guidance provided. The only way to be fair to all teams is to set it to the guidance.
True–that is why what I am suggesting would not actually do any judging, just extract pertinent segments for human review. Then that would be double-checked against the actual notebook when the scores are tallied.
Even if there is the occasional hallucination, it would still be much quicker than combing through the entire notebook. And it would not be subject to the all-too-human bias that prompted the discussion.
But I do appreciate your feedback.
Unless you can prove use of AI, the notebook stands. If it can be proven (to say hard to do is an understatement), it is a disqualification event that at a minimum goes through the EP. While my teams have suspected a few. The interviews usually disprove or in some cases fail to prove. AI is a valuable tool, but a totally bad crutch in schools. Interaction with coaches, teachers, professionals in the areas of the questions are better options. These options can also help students gain more confidence, learn what jobs they may want or not want, and the all to important human interactions. I too like to see otehr opinions. I just have distinct opinions on this based on things I hear about from one child’s college stories. The products being sent to me as a STEM professional from said colleges. The reliance on AI for answers has led to me newly hired people not to make it off the probationary period.
The performance on the day of the event should be accounted for. If the kids have a good design or have a great innovation, is it working on the day of the tournament? I have seen the Innovate and Create awards being given to the teams whose skills are in the bottom 10%. Is it sufficient if I write some innovative idea, or should the innovation be tested and proven on the day of the event?
The use of AI is tricky and can lead to a lot of unnecessary confusion. AI is a tool, a powerful tool infact. We want the teams to use AI to gather ideas as they brainstorm or debug a problem. I don’t think that is a violation of AI use. But if the students use AI to write the entire document (which I dont know how you one can find out), I do agree its a violation.
I have seen some really non-innovative innovations win. I am currently pushing hard to have the innovate award spun off the notebook “top tier” requirement as the rules governing the notebook grading are enforced differently it seems everywhere. The guidance out now suggests 10 minutes to deem the notebook worthy to get a in depth review. The in depth review should take about 40 minutes. Personally I will have my judge teams following that regardless of size. Holding the times consistant is the only way to have a fair grading. This is also why I have yeet to switch to remote grading of notebooks. I have yet to be tod how to maintain the controls needed to ensure fairness. So this year will be interesting.
The innovate IMO should get a state qualification as if it is actually innovative, it would be some of the most STEM thinking in the design. These teams are capable, but with the push to win, seems most have similar robots. The innovation makes them more unique and funner to judge. While I continue to push for that, I think have the JA’s read the Webster’s definition of innovation might help. Some of the winning ones versus ones I have seen…leave me highly disappointed.
Feel free to email VEX to push for better Innovation Award criteria. I do a few times a year.
The use of AI is tricky and can lead to a lot of unnecessary confusion. AI is a tool, a powerful tool infact. We want the teams to use AI to gather ideas as they brainstorm or debug a problem. I don’t think that is a violation of AI use. But if the students use AI to write the entire document (which I dont know how you one can find out), I do agree its a violation.
Sometimes they make entirely to easy with the cut and paste. Other times during the interview the way items are discussed versus the notebook are not close. The later is not enough to DQ, but does impact the discussions. Hence why interviews and understanding are vital. AI is only a tool, but as someone in a technical field, the wording and way a question is put to AI drastically impacts the answers for engineering, science and likely most STEM, I would take a few grains of salt with the answer as most are really basic and cookie cutter. Which is why I would use it for maybe a patch, but reach out to industry to get better answers so the teams maximize the learning.
How would AI be able to help in a meaningful, legal way?
AI can provide ideas on improving terminology, ideas for concepts when stuck for inspiration, ideas for code. Just as a tool to improve your team. Note: AI does not always provide best solution, but through process of elimination, can speed up the troubleshooting. Just do NOt have it write your code or notebook pages or slides.
Well, we need to follow the judge’s guide and trust your judges. I don’t know what else to tell you, especuially as we have a new guide this yer for Global.
Many experienced judges insert their own processes into the judging process and weather or not that is appropriate can be taken up with your JA or in the Judging Q&A.
There should not be a “This is how we do it” way.
I agree that the guide is the rules. Wrangling the judges can be an adventure. I am hopeful that this year will be better for clairty and consistancy. More to follow I will bet.