Judging Criteria - user response

(This is a response to the thread “Judging Criteria Value” in the Online Challenges Official Q & A, which I can’t post in).

Despite official rules/criteria for the various challenges, beccause 4-6 of the finalists in each category are chosen by user voting, there are bound to be finalists that don’t fit the criteria.

In general, a public voting system works well when:

  1. Most of the voters are personally unaffiliated with the contestants (e.g., American Idol), making the voting relatively impartial.
  2. Casual voters (who have a low investment) don’t have to spend much time in decision-making.
    It DOESN’T work well when there are many contestants and/or complex criteria that require careful weighting of points, as casual voters don’t want to spend 10 hours trying to give fair evaluation to 100 contestants in a category.

A judging system with complex criteria works best when every person who judges is committed to upholding the criteria and evaluating EVERY contestant, not just committed to carefully looking at one entry. This would probably only describe the official judges.

Because of this, my opinion is that official judges should determine most of the 10 finalists. However, to encourage interest and outreach, I think it would be appropriate to give 1 or 2 “voters choice” advancements to the most popular choices. To avoid “down-voting”, having a binary choice of “like” vs. “leave blank (i.e., don’t vote on an entry)” rather than graded scale could be used. Popularity matters some, and those who enlist friends and contacts on Facebook serve a purpose, but I don’t think popularity should matter as much as it currently does. Having a less complicated option (“like” or leave blank) means that viewers are also more likely to look at and possibly vote on more entries, not just that of their own team.

With my students submitting an entry, I realized how difficult it is to be impartial. Trying NOT to deliberately downvote other entries, I started out only voting if I could give a “good” vote. However, a SNAFU arrested the process and in the interim, I realized that like the Pirates of Penzance who always lose because they refuse to fight a weaker enemy, I was setting my team up for failure. My inertia (i.e. laziness) was justified by pleading integrity. I thought of voting in another category, but the sheer numbers of entries were overwhelming, especially because I wasn’t as interested in the categories where my team didn’t have an entry. Interest and fairness were working in an inverse relationship.

xraymypanda, I’m also disappointed that my team’s entry didn’t make the finals – being in the top 10 is a kind of acknowledgement, even if you don’t end up as one of the 3 “winners.” In a previous year, it’s likely we both would have advanced, but there were so many good entries this year. Your essay was my favorite and very fun to read, even though the system wouldn’t let me leave a comment.

I think the big factor this year is that originality. The judges were so tired of the standard promote montage that anything that deviated from the ‘title-picture’ format immediately stood out (especially with 54 entries). That’s probably why some good videos didn’t make it to the finals (like the sweetch video), and some videos (like one that is four minutes long) did make it to the finals.