Digital notebook advantages - Potential Online Judging System

VEX IQ Robotics Competition Discussion General VEX IQ Competition Discussion
1

There is no judging tag… weird…

I think it would be really great to have some online portal where you could submit your notebook and have it judged. (This would just be the front of the rubric, the 45 points) For it to work well, there would have to be some kind of fee involved but I think it would be really helpful.

As an EP, I could collect the notebooks digitally by Wednesday before the event and then submit them to the online system. I would get the notebooks judged and then my judges on event day would have those scores to work with already and all they would need to do is interview the teams. (I’m not sure how helpful this would be as they would have to read them anyway, but I’m hoping that it would give them a jump on their work.)

As a team, you could submit your notebook to the portal at any time to see how things are going for your team. Lots of new teams don’t exactly know how well they are doing and having a place to get their work evaluated can be really helpful to push them in the right direction.

We always tell kids and coaches to “use the rubric” but something a little more concrete I think would be really helpful.

There are a number of things you could do to increase reliability. You could have two judges on each notebook and if their scores are really far off then they could give it to a third judge. Judges who consistently are off the mark of the other judges would be removed, etc. There are lots of systems that are good that we can get all those kinks worked out.

I don’t know but I’m guessing that there are different amounts of pages I’m thinking there would be some kind of size surcharge. Maybe $A for the first B words and $C for every D words after? I’m not even going to put a number because some people get fixated on that.

There could also be an added system where you could do a mock interview as well. Most teams have very few opportunities to get interviewed and a system that would allow them to practice the interview with some impartial party would be pretty helpful as well.

As an EP I could spend some time judging notebooks in the system as a credit so when my tournaments come around it won’t cost me a thing.

Also, RECF would need to continue to allow the online system of digital notebooks into the future for this to be viable. I think that this would be part of a solution to help us get consistency across events, which is something the RECF is always harping about.

8 Likes
2

With the option of online notebooks, which some teams were asking for years, that is finally here, it is certainly a very tempting idea to take advantage of almost universal online presence and utilize judging resources from around the world in a load distributed manner.

However, in my observations, the notebook judging by volunteers at the local events is very subjective. There are only a few experienced judges that have been doing this for years. The only thing that makes it relatively fair is that at each competition the same set of judges looks at all notebooks, comparing them to each other.

Any online system that cannot guarantee perfectly objective grades by any random judge, needs to either rely on large number of independent judges looking at every notebook to achieve objective average or mimic a local competition, where the same set of judges comparing the same set of notebooks for a particular event.

In my experience, it takes very little time to sort the notebooks into A,B,C piles and then most of the time is spent to discuss the few notebooks in the A pile, which are on the short list for Design and Excellence awards. These discussions are then very helpful to shape the way in which teams from the A list are being interviewed in order to determine the winners.

I definitely agree that for some events, where it is not easy to find experienced local volunteers, having online notebooks and zoom interviews allows more flexibility in recruiting judges.

I think that one of the most important functions of notebook review is to provide feedback to the team that is more detailed than a small set of numbers. Grade for each category is important, but the detailed feedback will be much more helpful for the team to educate them and help them improve their notebooking skills.

It would be a fascinating experiment to develop a system which provides objective grades via averaging multiple reviews and then calculates internal weighting coefficients for each judge to determine how expert is their grade in each category or area of expertise. However, I am not sure if RECF is ready to invest in such experiment.

Perhaps, they may be able to partner with a university which is interested in this type of research using multiple teams’ notebooks and volunteer judges and the realistic subject of their experiment.

3 Likes
3

As a program organizer for our school, I do wish there was a feedback system (other than winning/not winning) an award so the kids could learn where they are doing well and also learn the areas that require improvement to bridge the gap to the best in show.

I get that judging is ‘private’ but a few notes like “you didn’t include any code” or “add your table of contents” etc could teach the kids where they need to improve.

4 Likes
4

Can someone explain in detail about this?
— Will the digital logbook be consider finished by the time Google form has to be uploaded? For us the deadline is 3-31-2021. We understand a logbook link will need to be listed on the Google form so it can be viewed by the Judges.
— Can the logbook be updated from April-May with the latest information of the logbook be uploaded for Worlds in May?

5

I’m not sure that this is the best way for cost. Well-written, longer notebooks that accurately document the robot design tend to be longer, especially near the end of the season, so it would mean that the better written notebooks would cost more to get feedback on. The rest of the Idea looks nice, though!

1 Like
6

It sounds like you are talking about a particular event. This is not the place where you will find information about that. I would ask the event partner for your event.

1 Like
7

Steve Rummell instructed me to put this on Vex Forum to get an answer from GDC.

8

The GDC will not give you any answers here.

If you need the judging Q&A, it’s here.
https://www.robotevents.com/judging/2020-2021/QA

If you want to post on the forum so anyone and everyone can chime in, click on the “+ New Topic” button.

2 Likes
9

Also this question was answered on the judging Q&A within the last couple of days:

https://www.robotevents.com/judging/2020-2021/QA/768

1 Like