I’m hoping I can add to this thread, without repeating too much of what has already been said. I’ve volunteered as a judge at Worlds twice and like to think I have a fairly good idea of what’s happening. The format they use at Worlds might be different to other competitions just due to the overwhelming number of high quality teams, but I think the process should scale down well. I should also add that I’m no expert and I’m only commenting on what I’ve observed.
First of all of like to say if you get the opportunity to judge at Worlds, take it. It’s a deeply informative and rewarding experience.
While teams are checking in, design notebooks should be handed in. Before the competition starts is the only time it’s quiet enough to go through them and judge them. Some competitions also set a deadline to hand in notebooks by, (up to a week before) this just allows judges to have a proper look through the notebook before design interviews.
Design Notebooks are judged from the design award rubric. (Which has already been linked in this thread). The rubric was rewritten recently to try make judging notebooks less subjective, but, of course, it can never be written out completely. You tend to find that opinions will normalise as you do more and more notebooks. Generally you have a set of judges (two judges) working on a notebook at the same time. All the judges can do this, not just those doing design interviews. Having judging sets also helps standardise the rubric scores. As notebooks are scored they are organised into two piles. Those above a certain threshold, and those below. Those above the threshold will be given design interviews, those aren’t didn’t make the cut. This is purely because of the time limitation and because design award is looking for the best notebook/interview. Design judges at Worlds now go to pits to do the design interview. With less teams to interview it means they have the time to go over each notebook in more detail after the initial judging process.
There are also Excellence judges and pit judges. Excellence judges at Worlds have an interview room and allocated slots for teams to be at the interview. Excellence doesn’t only take the interview into account, there is an excellence calculator on the REC website which outlines all the aspects that are taken into account. It is still a judged award so the top team on the scoresheet may not be deemed by the judges as excellence worthy as another team. But it is of course up to the judges to decide.
Pit judges (what I’ve always done) are the ones who walk the pits and ask questions for Build, Create, Amaze, Innovate and Judges awards. There are no set questions for these interviews and each set of judges is judging all 5 awards at once. The scoresheet has a small area to give team marks in different areas for the awards, but it’s best to write in details and take a photo as a reminder. As judging sets do more interviews they tend to get a good idea of which teams are better than others in certain aspects. Each judging set can nominate teams for any of the awards. At Worlds this is done by writing their team name on a post it note and sticking it under a sign for that award. Obviously they only nominate the best teams they’ve judged. In terms of identifying which teams have already been judged, a simple sticker on the pit sign is used. When all teams have been seen once, judges should reconvene to discuss the nominated teams (probably about 6 for each award). Each judging pair gives the reason they nominated that team and those nominated teams are seen again by a different judge set. Judges reconvene again and teams are ranked as they are discounted for the award.
The reason teams are ranked is in the chance a team becomes ineligible. This could happen for a variety of reasons. Also being well judged by pit judges helps teams towards excellence.
Hopefully that covers the process. Tarek has released a whole load of useful information which is on the REC website under EP documents and then judging documents.
Shout out if you have any questions.