We had 32 teams at an event this weekend, middle school VRC. Neither the Innovate nor the design award were given out. The JA said no notebooks met the requirements for either award to be given out. I support the JA and this isn’t a dig on him at all. I had 3 teams compete and all 3 had the innovate submission form after the cover page. One of my teams has won innovate and design awards this year with the same notebook. We are very confused how their notebook now doesn’t meet the requirements. Obviously the JA can’t give feedback, so we are left in the dark about why the team isn’t eligible, but they won both awards previously with the same notebook. Here is a list of what we know:
The form was inserted after the cover page
The form identified the pages in the notebook where the innovate info was covered.
The info about what is innovative only covered one thing, using an AI sensor to help drive a color sorter for rings on the intake.
This team was the only team with a color sorter at this MS event.
Are there are JA’s out there who might help me guide my team in this matter? I have no idea what advice to give them so they are eligible in the future.
The fact that no teams were eligible for the design award leads me to believe that each notebook must have something in the emerging section on the rubric. I want to post the link to the notebook so people can see, but it’s a live notebook and one of the best out of my 9 competititve tournament teams this season.
Have you considered reaching out to the JA to see if they would be willing to do a notebook review session and/or mock interviews with your team(s) at a practice? The judges cannot provide feedback or details about judge deliberations at an event, but there is nothing prohibiting judges from volunteering time outside an event to go over things with teams to help them improve. You would want to make sure not to put the judge in an awkward spot by asking questions about the past event, but asking for and receiving input on a notebook or practicing interviews outside of an event would be OK.
Are you a JA or do you have judging experience? The team doesn’t want their notebook to be shared, but we really want feedback and not sure where else to get it.
I have judjed events before and would be willing to take a look at the NB.
some reasons why JA may have said no NB meet requirments and others dont.
The JA may have felt that none of the NB’s meat the “developed” requirments for a notebook and therfore was not eligable for an award.
Some JA feel wrong about not handing out awards so they bend the rules and let subpar NB’s go throug hjsut for the sake of handing out the tropies(im in no way saying that this is what happend at previouse tournaments)
Pure speculation, but given that judging is subjective by nature, perhaps these judges graded notebooks more stringently than the judges from the prior event.
Turn the frustration over not winning the award into something positive. Go thru the rubrics and make it easy and clear for judges to tick each item off the rubric. Play devil’s advocate with them - for example for the color sort how is it “a key part of their team’s robot design or gameplay that is in use at the event.” Again, pure speculation w/o context. Do they have metrics for how many rings their color sort filters out per match or skills run?
Here’s my perspective (Judge and JA for local, state, and worlds)…and here are a few some general comments (in no specific order) without looking at your specific events, details of design, or region:
Every event is different, and the events change dramatically from early season to later season. Something that might have been innovative early on might not be seen as so innovative later.
Every notebook panel is different, and methods of scoring may be different between events (especially differences between physical and electronic notebook scoring). If the notebook review judges missed something, this event that was noted at the last event, the end result could be different.
No consideration is given to a team’s award history (according to the Guide to Judging). But I think your point was more that the team had a “winning design” and was surprised not to win again, as opposed to being denied an award because they had won it at an earlier event.
The makeup of the judging panel varies from event to event. If the panel has experienced robotics people, their take on “innovative” might be different from a judging panel pulled from industry, where what your team thinks is innovative is actually commonplace. Conversely, judges inexperienced with VEX might be impressed by something commonly seen on youtube videos, if they have never seen it for themselves.
If several of the judges were pulled from industry, they might be less impressed (or even put off by) nonstandard terms often used by robotics students, like “lady browns” and “pistons”
The biggest single factor, most likely, that is often not taken seriously enough, is the interview. Just like getting a job, the notebook (resume) gets you the interview, and the interview is where you have to sell yourself (or your design/innovation). Interview prepration and practice are keys to earning judged awards.
The best way to help your teams in the long run would be to volunteer (or get some of your teams’ parents) to volunteer as Judges, to learn the judging process first-hand, and become advisors for your teams to improve their process and interview skills.
Yes, we have thought about this. We have a parent who judges every tournament he can as our judge helper. He runs mock interviews for all my teams. He was actually a judge at this event, but to avoid a conflict of interest, never judges any of the our team’s notebooks or interviews.
According to the details under the design award, they don’t need to be high quality, a 2 on each of the 1st 4 categories on the rubric would suffice: * Be at or near the top of Engineering Notebook Rubric rankings with a Fully Developed Notebook. All notebooks with a score of two points or higher in the first four criteria of the Engineering Notebook Rubric should be considered Fully Developed as this would outline a single iteration of the Engineering Design Process.
Nobody is frustrated, just confused. We have gone through the rubric forwards and backwards. I’ve had multiple previous judges use the rubric to evaluate the notebook and they all agree, at worst they would get a 3. Yes, they provide 3 ideas, reasoning for choosing the solution they chose, and data supporting evidence of testing.
Based on years of experience I would agree with the above posters that you, probably, happened to run into a set of judges not familiar with current season VRC specific terminology (maybe even slightly confused by it) and applying their subjective prior industry experience to award selection criteria exactly as written in the rubric.
While unfortunate - that’s the uncertainties of the real life that make it in one form or another into the game domain of VRC.
However, lets consider an alternative.
Would you rather prefer to be judged by a soulless artificial intelligence engine that will uniformly apply same judging criteria derived from some secret training set curated by RECF to all the teams in the world?
Could you guarantee that it has no bugs that will drive everyone crazy?
I’d say help your team learn how to enjoy subjectively human aspect of interacting with the real judges, while they still can, and work on polishing they interview skills to amaze and dazzle both seasoned engineers and causal volunteers who never heard of VEX before.
Different JA’s have different protocols, values and their viewpoints are driven by different experiences. That my kids do understand. They are aware that different judges have different opinions and try to do their best with professionalism and a good sense of humor. But…it’s my job to provide some sense of direction and guidance, so that’s what I was trying to do.
Oh, and definitely not a soulless artificial intelligence engine with secret training, we preferred the AI with souls for sure.
The other thing to consider would be G1 and G2 violations that would take you out of the running for any judged awards but you wouldn’t get the feedback to know it had happened.
Let me off road for a moment. (This is kind of a rehash of one of @kmmohn points)
I’m going to ask “Is color sorting innovative?” There has been color sorting since the early days of VRC and VIQ. In pretty much every event where there are multicolor game pieces, there are color sorters. The VEX library even has a section on how to do color sorting.
The camera (I loath to call it AI) has code in it to tell you the colors that it’s seeing. Your side of the code has got to be 10 lines or less. It’s not innovative, you are using it for something the sensor was designed to do. Would using the rotation sensors to find a distance be innovative?
Now maybe if you had a highly polished aluminum plate that reflected the light from the object to the camera because there was no space otherwise to mount the camera, I may go innovative way to cram things together in a cramped space.
The pendulum swings back and forth and sometimes it resembles a wrecking ball.
I don’t have much new to add other than I would guess you had some over zealous judges.
The design notebook is the hardest part for my 1 HS team. This year was the first one where they have even made an effort. While I have not been involved with VRC long, I was an engineer for 12 years prior to teaching. Technical writing is difficult even for degreed engineers, and if you had industry professionals, they may have been venting some on the sorry ability of our school system to provide the basics of technical writing.
One thing I think may be needed, are examples to go along with the rubrics so that judges have an idea of what makes a DN “well developed” or not. The AP exam scoring guides always have examples of what makes a good answer, and what does not. (If this exists, please excuse my ignorance)
I would also add that students should be allowed to crawl a bit before they are asked to run, and awards should always be given to whichever team does the best at meeting the requirements for that award. Students are competing against each other, not an arbitrary standard that VRC sets, regardless of the intent of the rule or scoring change. This is especially important at the MS level. (IMHO)
The JA in our area doesn’t let the judges see the Innovate submissions. The JA goes through them, talks to the refs to find out if the refs think the submission is unique and not in use by any other team. If they say no, the judges don’t see the submissions at all.
So your area is not my rodeo, so I’m not in a position to judge them.
But if that was going on in my area I’d have a serious sit down with the JA. As a JA we are taught “Don’t make things up”. The judging guide does not say anything about using the referees for those awards.
It’s a pretty serious allegation. But the proper procedure is to have your Mentor/Coach contact the RSM in your area about it.
(As a general FYI “Don’t make things up”, pretty much also goes for the Referees and Event Partners. RECF admires initiative, they don’t admire not doing what the rules / instructions say to do. )
Let me preface this as even though I help out in the event process and have a reasonable amount of knowledge on the EP side I am not a EP as a student, so take this with a grain of salt, however I think I have a reasonable idea so I’ll put it out their for someone to fact check me on, I’m training myself to hopefully be able to do some more EP, judging and refing work, but I hope I get this right.
From where I understand it that JA is breaking conduct, tread waters carefully and make sure you are 100% sure of everything, then have your mentor talk to the RSM.
That the "problem you’ve Identified, however the solution needs to be taken a bit more carefully IMO because coming out guns a blazing to these staff of “they did this and that’s wrong” can end badly for you. Be careful the way you attack things though, even though if what you said if correct the JA probably won’t love it, should that affect you decision to report them, no, should be be careful of coming of too strong and blaming them too quickly yes. You may try to approach the RSM in a way of “Hey, I believe that the JA at this event did _____(to come off softer you may add “although I could have mis read the situation”) , could you maybe look into a clarification as to what happened”. This way you don’t attack the person or accuse the person too quickly, this let’s the RSM verify for you to make sure that you were correct, however it still is asking for something to happen. I know that a lot of people will want to come off harder because it is a vary serious offense so they want to address it but I think that leaving it in a bit softer and kinder terms to the JA will help your case so you get seen in the “I want to make sure policy is help” light and not the “I wanted to win an award and didn’t so it’s your fault” light because while it may not be professional on their side to judge you like that subconsciously it will still happen.