Thank you, Dave_Flowerday for the reminder to bring this back to focus.
By way of compromise… would posting the top 5 contenders for Design Award, without scores etc, be better (knew you were contender and on the right track or really need to reexamine what you are doing) or be worse (why was I second?)?
Coming from a State that’s super competitive for the Design Award, I guarantee arguments and fights would ensue.
Thanks Dave. As should be obvious from the list of things I put in my post, TM was the probably responsible for NONE of the issues I describe. It was all simple infrastructure stuff, and operator issues.
Are teams arguing and fighting at each tournament now when they don’t get the award? As a mentor I’ve left tournaments scratching my head about some of the judged awards, but I’ve never gone to the EP or Chief Judge about it.
Please no.
It would not help. It would cause more tension for every other team that was not “top five”.
Generally speaking, I would think that the higher ranked teams already have a pretty good idea that they are in the top 5-10 and they really just want ideas on how to be the top 1 (or top 5 at Worlds). If you are surprised that you won or placed high for Design or Excellence, there is an issue. It is the people who DIDN’T place high that need the most overall assistance and maybe would benefit from knowing what stopped them from ranking in the top, IMO (“here’s the part of the rubric you should expand on” or “would have been top 5 if you’d added xx”). The ones who DID place high likely would benefit more from very specific recommendations (“add xx to your intro” or “put in more labeled diagrams”) rather than a generic “yup, you placed high”. Regardless, a PRIVATE note / comment would be far better than a public listing.
Knowing the other top contenders in a specific (“these are the team numbers”) way could lead to many searching those specific teams out in an attempt to talk to them and take a look at their notebooks (so as to get ideas on how to improve). Not a bad idea theoretically, but I can easily see an evaluation of the notebook leading to questions and complaints about why a particular team might have placed high (“strange that xx won or yy was in the top 5 when their notebook seems fairly bland / lacking content / wasn’t updated / etc”). Even knowing that more than the notebook comes into play for the top awards, it is hard not to question things if you see a major issue with the notebook (such as hasn’t been updated for 3 months or reflects a robot that they didn’t actually bring to the competition). That is especially true if there are other notable issues (such as the Excellence Award winner didn’t do Robot Skills and barely squeaked into top 10, or it seems that a mentor did the building / programming).
I could personally disagree but maybe I’m just hard on myself. This was at States with 46 teams, 15 of which had won Exellence, Design, both, and some teams even had multiple design awards. My team had no idea we were in the top 10-5 for design until a few months after the competition when I switched teams and said my notebook last year was terrible(still kinda think it is formatting wise) and was told the coach and a teammate(was on a sister team last season) heard the judges talking about how my notebook was ranked higher than the team(also a sister team) who won design award. Finally realized why that sister team tried to hard to get me to join their team.
I would have to agree that this is a great idea theoretically, but like the problem you suggested there could be another problem. the sister team I was talking about earlier has gotten many design awards and other teams knew this. This then caused a coach of a team to steal their notebook one time because the coach knew they had a great notebook even though the coach had been told he could not look through their notebook.
I completely agree. Lower ranked teams wouldn’t benefit from knowing small things they missed, but instead what large areas that didn’t have a lot of focus but should have where as higher ranked teams probably covered most of the large areas and just need help with small things that they may have missed, not know to include, or just need more of in the future.
OK, I’m convinced the posting a top 5 would not be helpful, though I’m not sure how it’s different from providing feedback if a team is prone to arguing.
Back to more brainstorming 
Im getting unclear on what the purpose of the feedback. When the thread first started I thought the feedback was to help teams apply the engineering design process. Now as the thread has progressed I am getting the sense it it so help teams that are not winning the design award win design award. The type of feedback given would be entirely different in those cases. And I guess I’m confused as to why coaches can’t handle a lot of this.
I think the main differences lie in private vs public and specific vs vague. If you post the top 5 for everyone to know, anyone can start debating the merits of those teams but all they really know is that OF THE TEAMS PRESENT they stood out for some unlisted reason. If you provide specific private feedback to each individual team (or coach), then that team can discuss only their own feedback and compare it to their own activities.
I see those two as going hand-in-hand, to some extent. The notebook and interview are supposed to show how well you are learning and applying the engineering design process, and the award (or high ranking for such) is direct feedback to how well you are demonstrating that knowledge. If you feel that you understand the design process fully and that you have created a notebook that clearly sets out your understanding but you never win an award (and you have no idea why), then you have to start asking yourself if you really understand and if you are applying it correctly / fully / clearly / etc. Without feedback from mentors / coaches / EPs / other teams / judges, it can be hard to see where you might be having issues.
But the same situation already exists regarding the Design Award Winner. Everyone knows who won, and no one, including the winners, knows why.
You can always give out a Judges Award. “The Judges were impressed with this teams design notebook and they really did a great job in their interview, the award goes to Team 23T” Or some such wording. Kind of like second place design award.
My purpose in doing the pilot and being such a “pro-feedback” person was to help teams do better with their engineering notebooks and with their interview. Teams get instant feedback on how well their robot works / doesn’t work during every match, I feel that they should get feedback on the other part.
If you are doing better on the notebook, you are now also doing better on your engineering design process. If you follow the score system, and do every one of the notebook things every time you meet, the fallout is your process gets better.
It’s a little backwards, but it works. It’s like getting students interested in STEM by building a robot. The robot is the catalyst. The notebook is the catalyst.
I don’t know anything about the competitions in other areas, but around here you pretty much know going into a competition the top few teams likely to win those awards and those are often the ones who indeed win … so discussion about the winners might be as simple as “man, they are always awesome”. I certainly wouldn’t say the winners often don’t know why; when we’ve won those awards, we generally know that they had a great interview, are strong contenders with the notebook, and are high enough in standings for Exc. We’ve only rarely been surprised at specific teams winning Design or Exc in a local competition over the years, and I don’t believe we’ve ever been surprised at one we’ve won (that is to say, when we’ve won those it was at times we knew that we were a top contender).
Now, if I were to see the top 5 or 10 in the rankings for the awards I may be more surprised at who is there… 
It is very similar in our area too. After one of the teams gets ahead in the notebook game they will get design award virtually at every competition they attend.
You can rebuild the robot or change your driving habits from competition to competition, but you cannot rewrite your notebook (at least you are not supposed too).
Because of this, most other teams tend to abandon any serious work on the notebooks once it is clear that one or two teams have a virtual hold on the award.
I asked Dan Mantz about this two years ago and he said that RECF is considering adding timeout period after team wins the award. For example, if you got Design Award at local level you could only compete for it again after two months or at States or Signature Event level.
This would encourage other team to put more efforts in the notebook and I agree with Foster’s logic that it should improve their understanding of what they are doing with actual robots.
I think only a small percentage of VRC competitors will end up writing actual Engineering Notebooks when they grow up, but virtually everyone will benefit from the habit of applying feedback based Design Process to their decision making, and this is what we are after.
If it was up to me, I would change Design Award into something more similar to multilevel scout merit badges, when multiple teams could earn them at every tournament by demonstrating certain level of skill following the process. Once you got beginner level badge, you are only eligible for next level, etc… Actual award would be given only at larger tournaments like States, Nationals, or Worlds.
I think overall, any feedback would be useful and appreciated by students and teams.
Some might use the feedback to improve their notebook because they enjoy doing the notebook and take pride in having a nice one. Maybe they just fill out the notebook for personal reference so they can better develop and reflect on their robot design and feedback could be used to make their notebook more organized and detailed for themselves. Some might be striving to get an award and need a tad bit of guidance. And some might be looking into careers that utilize an engineering notebook, and could use feedback in order to improve their notebook practices for their future. And for some teams it’s a combination of these. However, how the feedback is used will overlap a bit and the type of feedback given will be used differently in each situation and help the team reach their desired result. However, If a notebook is improved from feedback for any of these reasons, it will help improve a teams chance of design award, maybe just a little but that’s still something, even if that is not the teams initial intention.
This is similar to another situation I was involved in quite extensively. I created an elementary school science fair designed for the students/projects who were awarded 1st-4th place at their individual schools. This gave them an opportunity to compete at the next level . All students were given the rubric prior to the event so they knew what the judges were looking for.
We did not return the judges notes to the teams because they were both content and opinion based which of course leads to interpretation.
I’m not sure about this. In Ontario many of our events do “Academy Awards” style awards, with judges naming 3-5 finalists for the award, with one of those finalists being named the winner. I’ve always been a big proponent of this style, because it provides an additional opportunity to recognize more teams. It’s always seemed like a nice thing for more teams to go home knowing that they judges really liked what they were doing, even if they weren’t good enough to win. I found it especially encouraging for new teams or teams with lots of younger members.
I’ve yet to notice any additional tension or arguments at the events who recognize finalists over events who do not.
I think one concern I have with the phrasing was I thought it was a ranked ordered.
My main concern is that we provide a rubric but probably insufficient support for teams(coaches/students/mentors) to know how to do a proper job. Work in progress.
One issue with this idea is that, if a team won a design award at a tournament where the design award didnt qualify for states, the team would not be able to compete in the next event where design award could qualify them. I guess I’m all for finding ways to get teams interested in ENs but it is kind of like saying if a team wins a tournament, they cant compete for two months. Kind of feeds into the “everyone’s a winner” mentality. Like my friend, Lacsap says: if you want to win, build better robots. A corollary is: if you want the design award, write a better EN. Why dumb down the competition?
I understand your caution regarding “everyone’s a winner” mentality. If awards are less exclusive then people assign them lesser value.
But, in my opinion, having one team accumulating six design awards from almost every local competition also devalues the award.
I consider VRC Award System to be a tool (or a catalyst) that helps us teach students Scientific Method and Iterative Design Process.
If we design award system correctly, then, even if not every team will get an award, but every student thinks that it is within their reach, it will make the system more effective in influencing larger number of students.