Question about judging integrity and conflict-of-interest safeguards at robotics championships

VEX IQ General Discussion
1

Hi everyone,

I’m hoping to get perspectives from coaches, mentors, judges, and event organizers in the robotics community about judging integrity and conflict-of-interest safeguards at robotics tournaments.

At a recent elementary robotics state championship, I observed several patterns that made me curious about how impartiality in judging is typically maintained when the host organization also has a large number of teams competing.

A few observations based on publicly available event results and what I observed during the event:

• The host organization’s teams received 6 of the 11 judged awards (~55%) at the championship event.
• Across several tournaments hosted by the same organization this season, all Excellence Awards were awarded to teams from the host organization.
• During the event, many judges conducting engineering notebook reviews and team interviews appeared to have affiliations with teams participating in the competition.
• In one instance, it appeared that a parent or close relation may have been present during a team judging interview.
• While reviewing the results, it also appeared that many teams receiving judged awards had affiliations with individuals involved in judging, while several teams in the middle of the rankings did not receive judged awards before the distribution continued again with teams connected to the host organization.

I fully understand that judged awards are independent of match performance and are based on factors like the engineering notebook, design process, and how well teams explain their work during interviews. Strong programs absolutely deserve recognition.

However, because judged awards rely heavily on subjective evaluation, situations like this made me curious about how judging integrity is protected.

My understanding is that during judging deliberations, judges often speak on behalf of the teams they interviewed when discussing award candidates. If many judges have affiliations with teams from the host organization, how are potential conflicts of interest managed during those discussions?

I’d love to hear from the community:

• Are judges typically screened to avoid interviewing teams they are affiliated with?
• What safeguards exist during deliberations to ensure impartial decision-making?
• Are there clear guidelines about parents or mentors being present during judging interviews?
• What best practices help ensure fairness when the host organization also has many teams competing?

Maintaining trust in the judging process is really important for students, coaches, and families, especially at championship-level events.

I would really appreciate hearing perspectives from:

• Judges
• Event partners
• Experienced coaches
• REC volunteers

What practices have you seen that help ensure judging remains fair, transparent, and impartial?

I genuinely want to trust the system and continue bringing my team back to these tournaments year after year. Hearing from others in the community — including REC — would be very helpful.

Thanks in advance for sharing your experiences and insights.

2

In my experience as a coach and a judge, getting enough judges to offer judged awards can be difficult. Without discussing anything else, I’d say that if you want less APPEARANCE OF BIAS due to judges having affiliation with the hosting organization then the first step is to encourage more teams from other organizations participating in the event to provide volunteer judges.

3

I have been in the judging room at the regionals at US Create and at Worlds. I prefer not to be a part of the tournament where my team is participating.

I offer to judge at the events where my teams are not participating, and interesting enough, the EP will respond, “Thank you for asking; we have enough judges”. Of course, they do not have to take me when they have enough judges who are affliated to the home teams.

4

A suggestion for EP would be to hire Industry professionals or local university/ community college/ high school robotics/ STEM teams. In my opinion, kids hold high integrity and are less biased than adults.

If that doesn’t work and if you cannot find even 5-6 non-affiliated judges, I would suggest removing judged awards entirely from the tournament. The fact that you can’t find judges doesn’t mean that you create opportunities for conscious-unconscious bias, leaving us parents/ coaches/ teams having doubts in the system.

If you can’t find judges, you don’t do judged awards. As simple as that !!

5

Per JR2 in Guide to Judging:

Note: For any VEX Robotics World Championship qualifying events, Judge volunteers cannot have any direct conflicts of interest with any team at the event (for example: parents or other family members associated with attending teams or team coaches would be considered to have direct conflicts of interest; this is not an exhaustive list). This is highly recommended for all events."

So JAs and EPs for Regional Championships and Signatures must not allow judges with these conflicts. But if a volunteer fails to disclose relationships, I’m sure that can slip by. If a Championship is obviously favoring host teams, then the JA may be “in on it” anyway and your only recourse is to complain to your RSM and provide additional evidence to back up your claims.

In my years, I have resigned myself to accepting that the host org will always give itself a spot (sacrificial). I’m not even salty about it - usually the host org has a strong program with a number of worthy teams, and look, they put in a tremendous volunteer effort to make that championship happen. If a region has multiple strong orgs, you can also bet that each org will get a spot at the table too - it’s probably not a conspiracy, just a panel of judges with a lot of strong teams to choose from, and trying to please everyone. I’m sympathetic to that.

Having served as a JA in previous years, I prefer to recruit judge volunteers directly, and avoid “open calls”. That way I got people I know are experienced, and will be fair and intelligent about the judging process, who are passionate about youth robotics and even have some background that can be useful (work in technology, etc.).

I don’t know if you’re in my region, but here in IQ the Worlds quals are not exactly diverse either:

  • ES Judged Spots: 3 went to host or host-adjacent org, 1 to an indy team
  • MS Judged Spots: 2 went to host or host-adjacent org, 1 to an indy team
  • All other Worlds spots (8 in ES, 7 in MS) are taken by a single for-profit org (rhymes with WagiBids) - many with the same numbers, 6 of those via double-qualification (trickle-down cascade down to Robot Skills ranked teams).

Incredibly, out of 22 registered Worlds teams, only 3 teams across ES+MS represent non-host non-profit orgs. And one of those had to fly across the country to fight for their spot via a Signature. The Regionals basically kept everyone else out of Worlds. Unless you’re a (specific) for-profit org team or with the host, it’s virtually impossible to get a spot. Welcome to VEX IQ.

Don’t get me wrong, I think teams overall had a great time at the Championships, which were wonderfully run, at a great venue, with high production values. I commend the EP(s); that’s a tough gig. But RECF needs to take a look at what happened and (a) seriously consider allocating some waitlist spots to improve diversity from the region and also reward highly judged teams (e.g. Design or Innovate winners from Signatures), and (b) look into what’s broken about some of the regions. At the very minimum, I think the double-qualification flow needs to be re-examined (for IQ anyway).

For example, why do extra spots trickle down to Skills? What if double-qual fell back into the awards sequence? For example, if a team wins Robot Skills and Teamwork (common!), then the double-qual spot can trickle to the next judged award. Of course, there’s still the potential for issues with judging (now biased judges would be motivated to stack all judged awards).

Just fixing this flow alone would “fix” 4 of the 12 ES spots and 3 of 10 MS spots and greatly improve diversity, and I’d argue, quality, of the cohort going to Worlds. It’d also probably adjust for the perverse incentives for elementary age kids (and their orgs) to push for these world record shattering skills runs which I feel is counter to the educational (and often, G2) mission of robotics.

Sorry, got a bit side tracked… our teams and many others I know go through this every…single…year. I hope your region is better.

6

I’ve been a coach, a judge, and a JA at a variety of IQ events and seen a ton of issues related to this. In many ERCs I’ve been at, let’s just say that the slate of judges represented a small subset of the competitors, either because of direct ties or because of clearly observable traits. It’s been defended because “there was a public link posted so we just take who we can get” but it’s really pretty egregious. I’ve seen a judge hand their phone to a parent of a kid on an award winning team, hold one side of the banner along with the team holding the other side, and then do a cheeky smile and get their phone back from the parent. Just nuts…

The guidelines are clear- the COI restrictions were strengthened in the last year. I suggested to EP/RSM a couple of things- (1) Have the slate of judges stand on stage at the drivers meeting so the kids can see them and also there’s some accountability there [This happened at the immediately subsequent event but not any since then] and (2) work with groups from around the region to recruit judges [This definitely has never happened]. I appreciate that #2 would require more work in some ways, but I guarantee that different parts of the region would be happy to recruit a few qualified judges.

My first year as an ERC judge, an alum of a large program spoke up after those that interviewed a team from that program said the team hadn’t addressed a topic and the alum said “I’ve been working with them all year so I know they did XYZ” and the JA just let it go. At the heart of it, there is a lot that comes down to what is said by whom in the room. Recruitment simply must be from diverse and/or objective sources or it’ll never be fair. And short of an EP/RSM being in the room, no one will ever really observe these egregious processes.

I’ll also be the first to say that I’ve been in plenty of judges rooms where Excellence literally had one viable candidate, and then for Design there was only one other team with a developed notebook, and then there was only one team that filled in the Innovate form. So is work to do on the student/coach side.

But my overall gripe is just that judging is a black box where students get no feedback. If you filled out a notebook, had a novice coach that couldn’t give strong feedback, and then didn’t win an award and didn’t know why, it’s clear that a lack of feedback kills any design process- most students I know either prioritize the notebook, win early, and keep going or give up almost immediately. We try things like rewarding best “X” page or other things at league events, but it’s just not going to be a priority of a resource-strapped public school elementary team.

7

I like some of the ideas I’m reading here, and will re-phrase + add one or two:

  • Judge Transparency: Driver meeting should introduce each judge, and then they are free to return to their duties. Teams can raise concerns about COI directly to the EP or JA.
  • Remote Judges: For all Championship events, judges should not be from the same region or any competing organization (recruited and matched by RECF, similar process as Worlds remote judging)
  • Judge Feedback: Have an anonymized mechanism where judges can provide some feedback to teams, and that feedback shows up in RobotEvents (near the digital notebook link) - how do teams/kids improve without feedback? Teams can go year after discouraging year not realizing some fundamental mistake they’re making on notebook or interviews.

Some other ideas I’d advance:

  • RECF JAs: Provide the JA for all Championship (with Worlds spots) events. Regional and small events would get a remote RECF JA. Some of the cost of those events can be increased to pay for these roles if needed. I know my teams would be happy to pay more if it can help ensure fairness in judging! Most Championships have enough teams that even a modest additional charge in registration should be able to cover a RECF JA for their time. Surely worth the cost of a sandwich.
  • Top Robot Photos: Judges should receive clear photos of the robots of Robot Skills winners of top events of the season. For example, the top Robot Skills winner could be required to have the robot photographed by the JA/EP for submission to a RobotEvents page / TM screen. This is especially important to capture the early season China/Asia tournament winners. All Judges will be able to view the photos during deliberations (but may not record or take copies). This helps with the problem of judges being educated about the originality and potential provenance of robot designs, and thus help spot dishonesty about designs.
  • Double-qualification Cascade: My earlier proposal: double-qualification should cascade not to robot skills, but to prioritized awards sequence
8

Thank you for sharing this perspective. I really appreciate hearing from someone who has served as a coach, judge, and JA across multiple events.

Your point about judge recruitment from diverse and independent sources really resonated with me. When the judging pool represents only a small subset of participating organizations, even if everyone has the best intentions, it can create situations where the process appears less impartial to teams and families.

I also agree with your observation that what happens in the judging deliberation room is essentially a black box for teams. When students put in weeks or months of effort into their engineering notebooks and design process but receive no feedback on why they did or did not receive an award, it can be difficult for them to improve. For younger teams especially, constructive feedback would go a long way toward reinforcing the learning aspect of the program.

I really like the ideas you mentioned around expanding the judging pool. A few possibilities that could help strengthen the system might include:

• Recruiting judges from universities or community colleges, particularly engineering or robotics programs
• Inviting industry professionals or STEM volunteers from outside the competing organizations
• Using remote judges for notebook review or interview panels to increase independence
• Coordinating judge recruitment across different regions so the judging pool is not dominated by one local organization
• Increasing transparency around conflict-of-interest safeguards during judge assignments

These kinds of steps could help ensure that judging remains both fair and perceived as fair, which is incredibly important for students and families who invest so much time and passion into these competitions.

At the end of the day, I care deeply about this program and want to continue bringing teams back year after year. Conversations like this help the community think about how we can keep improving the system so that students feel confident in the process.

Thanks again for sharing your experience; it’s helpful to hear these perspectives.

9

These are really thoughtful suggestions. I especially like the ideas around judge transparency, expanding the judging pool beyond the local region, and providing some form of feedback to teams. Even small changes in these areas could go a long way toward strengthening trust in the judging process.

For younger teams in particular, not knowing why they did or didn’t receive an award makes it difficult to improve their engineering notebooks or interview preparation. An anonymized feedback mechanism in RobotEvents, as you suggested, could make the program much more educational.

The bigger question I have is: how can ideas like these be formally shared with RECF so they can be considered at a program level? Is there a process for submitting community suggestions or proposals around judging practices and transparency?

I think many coaches, mentors, and judges would be happy to contribute to solutions that help ensure the system continues to be fair, transparent, and trusted by students and families.

10

While this isn’t specifically about judging, I wanted to share some observations regarding host-team bias in competitions as a whole. I’ll admit I’m likely biased myself because this affected my team directly, so please take my perspective with a grain of salt.

Our team recently competed in an event hosted by Host Organization where the referees were clearly affiliated with the host (one was even wearing a team shirt under their referee gear). Throughout the day, including the finals, we saw multiple instances where obvious violations by the host’s teams went uncalled.

This highlights a major conflict of interest. While I know it’s not always realistic to have a completely neutral host or volunteers with zero affiliations, seeing a referee in a team shirt while officiating that same team feels like a line was crossed. I’m curious if others have had similar experiences or if there are “best practices” to help alleviate some of these problems, even when volunteers are scarce.

11

That also happened to us last year. Funnily enough they didn’t do that tournament this year. It was a small tournament, so it didn’t really matter that much.

12

Regardless of the size of the tournament, let’s strive to bring fairness to the system. What are we teaching our future generation when the system is not transparent and fair?

13

I don’t think that the system is inherently unfair, it is just how good the Judges felt like you did in relation to other teams. Yes, this is completely subjective and can screw up double quals like it did in Nebraska, but also there are things that you can do to better your chances at getting one of these awards. My team win the Innovate Award at state and qualified for Worlds, but we practiced our interview a lot since we knew a judge advisor. Also, nobody knows what discussions happen in the judging room besides the Judges and Judge Advisors. A team could be from the host school, have a bad day, but can still win judged awards if they explain their design process well and document it well. I think that it is a good thing that the judging materials are destroyed afterwards, this protects the judges from criticism from parents or mentors from trying to argue that their team should have gotten a higher score from this other team when they don’t know what was said in the interview, and what pages of the notebook that was looked at.

As for the referee issue, since then the GDC has added (I don’t remember which rule it is exactly) the rule that the Head Refs have to read the game manual. The main issue with the tournament that we went to was that they were calling stuff based on a previous iteration of the manual.

14

Thank you for sharing your perspective. I want to clarify that my post was not intended to question the hard work of volunteers, judges, or event partners. I have served as a judge myself and truly understand the amount of time and effort that goes into running these events.

The questions I raised are more about how we can continue strengthening transparency and safeguards in the system, so that students, coaches, and families feel confident about coming back year after year.

For context, I come from a corporate background and have spent 25 years as a program manager, where structured documentation, process rigor, and clear communication are essential. Because of that, I place a strong emphasis on the engineering design process and documentation rigor with my teams.

For several years, our middle school teams maintained handwritten engineering notebooks, carefully documenting design decisions, testing iterations, and lessons learned. That discipline—and students being able to clearly explain their engineering process—has helped our teams earn invitations to Worlds and US CREATE.

When I started our new elementary team this year, I applied the same level of rigor. We regularly review the official judging rubrics, which are posted in our robotics lab, and students practice explaining their design decisions and iteration history—not just documenting them.

So while I absolutely agree that understanding the rubric and focusing effort in the right places is important, the questions I raised are really about process transparency and safeguards. We need to come together as a community to ensure the processes are in place so that students’ hard work is evaluated within a fair and transparent system that everyone trusts.

15

Hi, I may be a tad later on this, but I have reached out to the REC group for judging. Things I have noticed, being a JA in a small area. We cannot get judges that have no conflicts totally. So those judges do not participate in the judging or discussions involving their teams. Believe or not, my judges have been fantastic about this. The issue is not just a simple pool or judges, but the criteria being used to evaluate the notebooks and how the interviews can score or impact the rankings. Here is my opinion on one issue. The role and importance or the interview is getting downplayed in favor of numerical scores. So think about this. With the rampant use of AI (I am TOTALLY AGAINST USE OF AI or SOURCE CODE as it lessens the learning of the kids) the interview has in many cases revealed a complete and nearly total lack of understanding of coding and basic robotic concepts. These items in the events I am a JA at will impact the final rankings. So a team may score high in skills, but be hurt in the interview for lack of knowledge. Also, I am noticing that events using totally different criteria to go toward how long to allow a notebook to be reviewed. I preselect a time before evaluating any notebooks. The consistent application of time allotted to evaluate notebooks may mean small notebooks and or overly large notebooks score poorly. Part of the challenge is to write a clear more concise notebook and properly apply the use of appendixes. This year was horrible in my eyes for the whole consistency of grading and importance of the interview. I have thought about stopping, but I really want the kids to have a chance in the local tournaments. To bad the rules are not applied equally throughout. As someone that works in STEM, some of the really long notebooks while impressive, lack qualities that I look for as someone who has interviewed others for jobs. With AI JAs and judges need to reconsider how much they lean on numbers versus the interview. If a team scores well or extremely well and they cannot talk in detail about the design or code, red flags should be going up in your head about the team. Also, conduct issues should always be shared in the judge room no matter how trivial those observations are. Think about this if you have 50 teams and 4 negative comments that are more than a disappoint about not winning, do you really want to reward that behavior? I wish as a JA we could just post how many teams we had at the event, how many notebooks and how many positive and negative comments we get. So the coaches can better understand things. Consistency is the biggest issue I see.

16

I I will answer the questions in the order they are presented. I am a JA with 2years of experience and have judged for an additional 4 or 5 years. I am in a small area with limited resources. All of the judges avoid scoring their kids or teams they are close to. Most of the judges are harder on the teams they are close to and sometimes need to be reminded to be less hard. The safeguards for deliberations…You can ask them to leave the room, but I have not had a reason to ask them to leave as they normally don’t bother talking. If they do it is to point out areas to look at because of an impasse for the decision. Parents, mentors and coaches can be present at interviews, but must remain behind the teams at all times and are not allowed to speak with one exception…special needs may require the coach or familiar adult to interact during the interview. I see the last part as a reasonable accommodation and allow it. The last one is using the guidelines in the scoring lists and hold the times to the same. I wish more JA’s would limit the notebook review to a time. Talking to other JAs this is one area that really burns me up. The notebooks are allowed to be scored using different times even within the same event. So the JAs need to decide a better practice. I would love to see a JA forum where these issues can be called out and addressed better. Small areas cannot afford the 1 to 2 hours some notebooks are getting. I limit the times and the scores reflect it. Some of these notebooks are ridiculously long and the value added for the length fails to impress me. I am a chemist not an engineer. Only patent level experiments, high end research, or something hoping to end with a patent need to go long. If you look at the guide to judging Part 2 second paragraph (page 30 on the one I was looking at) it talks some about length. Some teams write a ton, but it fails to really add to the quality of the notebook versus the time required to read it. The balance has gone of the rails for longer is better. Not to mention AI being used. I have been at events were professional in AI and code were present. What they said about some of the top notebooks and all really reenforces the importance of strong interviews and weighting the interviews with the same weight as the scoring. As it stands, I will not judge again at an even where I do not review the notebooks or where the interview has little to no weight. It allows to much influence from outside, AI to be more specific. I can go on on the weighting and all. You do not need STEM people to judge whether or not these kids are on the up and up or trying to win at all costs. A few simple question about code and design will lead adults to conclusions. STEM is a better fit, but the others definitely add to the quality of the deliberations as they tend to catch the team interactions and question things STEM people may not but can lead to the whole…they really do not understand the robot.

17

As someone suggested we create a JA group for the Vex forum. I did email the judges to the “Competition Judging” [email protected] to ask for this to be created or who to ask to get it done. I want to provide the best possible experience for the kids, the judges and the adults. So keep making suggestions. I will do what I can to help.

18

Hi, I’m late to this. DRC, it’s hard getting through the wall of text. If you put blank lines between paragraphs it makes it much easier to read.

A few comments. Roboteers with special needs can have them. I’ve done interviews where there has been a support person, it’s fine. I know it’s allowed, but I can’t find it in the tomb of Judging documents.

Finding judges is always the EP/JA crying list. I’m sure your small town has people that may not be robot savvy but will make great judges. I have access to a set of car mechanics. They love coming to judge, they are blown away by what roboteers do with simple parts. Likewise, programmers / project managers that work from home. Try the “double step” I don’t want you a team parent to judge, tell me about your neighbor, co-worker, etc. Outside teachers, but even better is former / retired teachers.

Notebook reviews. This is hard, there are 7 page notebooks and 500 page notebooks. I train readers to “read every page until the first event”. Then “pattern match to the next event” are you seeing iterations? Next iterations? Are they building, are you seeing details grow?

It’s only the top 4 books that may need some detailed reading. Unless you are at a regional event, 1/4 of the teams didn’t submit a book. Another 1/4 of the books are 10 pages or less and you just need to stick a rubric in them to help them next time. The next 1/3 are good book, and will take a little longer. The last 1/6th are the hard ones. But you are not looking at hours and hours.

And I’ll touch on conduct errors. The JA needs to sort them out, in my 20+ years the “conduct errors” have been random and petty issues, only a few were worth looking at. I want to say as an EP I’ve been cited for conduct errors. “He moved the robot into the starting zone”. Yea, I did, I needed the match to go, the roboteers had no idea what the starting zone was, I showed them. Handing a spare battery to a team got me a complaint from a parent about “building the robot”.

Anyway my 4 cents. There is a forum that is gatekept by passing the JA test. Would access to that work for you?

19

@GomezSara @DRC @Foster

I posted the following on a different thread. But it seems more fitting here. I do believe that judging should be a major factor in who advances to State and World–but there needs to be major improvements to the process to make results consistent. Major work also needs done to mitigate conflicts that exist.

Reporting conflicts to REC doesn’t really work because in order to protect the business and ensure future commitments, they must protect their EP’s, Judges, and all the volunteers. This is understandable.

So there needs to be a solution where the “the law” if you will, cannot be circumvented by anyone involved—and if enforced, results in more fair and unquestionable outcomes.

These are just a few issues/conflicts our teams have encountered:

--An EP is a coach of multiple teams attending, the spouse is the Judge advisor, and their daughter is on one of the EP’s teams. The optics is terrible here, and you can guess the backlash from the audience when the daughter’s team wins Excellence at their event. This is not a good situation and quite frankly unfair the child.

--An EP who’s got multiple teams in attendance (6-10) uses anywhere from 1-3 day old teamwork match schedule at the actual competition. When the other teams mumble and buzz about it in the pits, the EP announces that any complaints about a randomly generated match schedule is a G1 violation and any team complaining can be removed from the competition.

--The judge(s) has indirect ties to teams in attendance. For example, a judge who is a high school robotics coach and who just coached the EP a year or two prior, would have indirect relationship to the EP’s teams. However, when this judge is asked if he has any relationship with any of the teams, his answer would be satisfactorily “no”.

Here is that post:

20

It’s a problem, and what you described is the primary reason why I stopped going to events run by one of the local programs.

I’m a MS V5RC Event Partner. When planning an event, finding 3rd party, completely unaffiliated judges is my goal, but also, finding volunteers - especially judges - is the most difficult part of running an event. My primary concern with my events are that I run an event that is COMPLETELY above-board, and if I can’t find enough unaffiliated judges, I have two choices: 1) find a completely above-board JA to run judging and then ask each organization that’s attending to provide one person as a judge; 2) run the event without any judging.

I would recommend having a conversation with the EP regarding your concerns about the integrity of judging at their events, and that if that doesn’t change, you will take your concerns to RECF. You’ve got the receipts, but let’s face it, sometimes RECF won’t do a thing. Your alternative at that point is to run your own events.