In the last month or so coming up to IRI (Indiana Robotics Invitational) I’ve been doing a lot of pre-scouting, pre-strategizing, and re-familiarizing myself with the game. I’ve been spending an overwhelming majority of my time doing this on the on the Blue Alliance website (http://www.thebluealliance.com), and eventually I began to wonder why VEX doesn’t have a comparable site.
A little bit about Blue Alliance:
Blue Alliance is a website where you can look up any team and see what events they attended, their schedule for the events they attended, what awards they won, and how far they made it in the elimination rounds. Blue Alliance can also be used to look up an event where the entire competition can be reviewed similarly to what can already be done on Robot Events. But the feature that I think is the most useful that it puts almost all of the FRC matches recorded in one place.
So what do you think, would a Blue Alliance for VEX be useful?
I took a look at their scouting database (http://scouting.dhsrobotics.com/index.php) and it does have alot in common with Blue Alliance. 5327’s database has a lot of the same features as Blue Alliance, but there are still features Blue Alliance has that 5327’s database does not (videos, full match schedules, alliances).
How do they get videos for the Blue Alliance database? I feel like that’s probably manually done, which would take a major amount of work. When you say “would this be useful,” do you mean that you’re considering doing it yourself? Would be an interesting project, to say the least.
“Would this be useful” means would you find a site like Blue Alliance useful in any way, shape, or form. I would do this myself if my programming abilities didn’t stop at easyC, but unfortunately they do. If someone were to undertake this project 5327’s database already pulls data from Robot Events which to me seems to be the hardest part of the project. The rest of it should be layout and simple equations.
The videos are posted manually, but by the FRC community, not just the administrator(s) of the site, so it’s not as much work as you’d think.
Many teams put Toss Up and/or the event name in the title, and sometimes there are team numbers, match numbers, and scores in the description. If a program could find these keywords and compile a database of those videos, then you would have a lot of teams’ videos. You could never get all of the videos, but some is better than none.
I’m Josh Price, President of Gael Force (5327). We are currently in the process of making major improvements to our scouting database. We plan on finishing the updated version of the database and making it public by late August. Some of the additions we are considering are:
-Live tournament information for every VEX tournament, including full match schedules and alliances
-User accounts that teams can use to save specific teams and events, store their scouting data, and add more information about their team
-Different forms of ranking, such as OPR, and a comprehensive ranking that factors overall record, skills, and awards
-User submitted photos and videos for teams and tournaments
-Team submitted information about themselves
-Up to date world map of teams and tournaments
We are still in the early stages of development and are open to any ideas from the public. We want to make this database as useful as possible. Our goal is to give VEX teams across the globe an integrated scouting experience by combining raw data from RobotEvents with a user-friendly layout that eliminates the need for using RobotEvents for scouting. With this database, important information will be brought to the front of the website so users don’t have to dig deep within the site to find important information.
If anyone has suggestions or ideas for other utilities on our scouting website, please post them!
Have you considered incorporating TRSPs as an alternative to the official SP tiebreakers? The basic principle is that your strength of schedule is measured by the WPs of your opponents (except in the matches where they played against you) minus the WPs of your partners (except in the matches where they played with you). Normalising the TRSP score so that it is a proportion of the maximum possible is another option, which would allow for (crude) comparisons between tournaments as well as within them. Like SPs, TRSPs are only really useful for qualification rounds.
This is similar to Swiss tournaments in Chess and similar 1 vs 1 games, where the first tiebreaker is often your opponents’ match win percentage. It’s usually a better system than SPs in obtaining an accurate ranking of teams, largely because SPs are very easy for a winning team to manipulate so high SPs for a strong team often mean weak opponents rather than strong ones. The TRSP ranking has the additional benefit of giving a quick one-value summary of how difficult a team’s match schedule was.
We haven’t discussed incorporating TRSPs into team and tournament information yet. It’s a great way to determine the difficulty of a team’s schedule, and potentially the difficulty of the tournament overall, so I’ll definitely bring it up with the website team at our next meeting. Thanks for the suggestion!
The improvements you are talking about hit just about everything on my list.
If your planning on showing statistics in your database, in addition to OPR, you may want to consider a CCWM (calculated contribution to winning margin) and match average. You may also want to provide a best OPR, CCWM, match average, etc. for each team since most teams go through many designs and iterations during the season.
Yeah, so as Josh mentioned we are considering a comprehensive rank that takes into account OPR, CCWM, match averages, and other statistics that we think are important for a team along with displaying these statistics individually so you can rank teams by different statistics. We would also display the team’s performance at each tournament as well as the average statistics over the season so that we can balance out changes in the team and robot.
If I was on an FRC team, I think that thebluealliance would be an immensely helpful resource. A comparable resource for Vex would also be awesome.
But as was previously stated, the 5327 database is a great substitute. Although videos of the robot would be an amazing feature.
One thing that has bugged me on the 5327 Scouting DB is the winner column. Maybe bold the score of the winner (like thebluealliance.com) instead of a “winner” column. And when viewing one team’s profile, add a column, “won”, which would have a “√” or “X”. This would make it so much easier to see which matches were won and which ones were not.
If I could recommend something it would be to not include OPR or CCWM in any kind of ranking that’s intended to reflect a team’s overall strength.
OPR does have some value in finding the fastest scoring robots (which is useful if not all of them do driver skills). But it only works for efficiency robots that aren’t playing a descoring strategy.
I’m not convinced that CCWM is even slightly useful in VEX. The SP system incentivises small winning margins, so a team with high CCWM is probably just deliberately not going for SPs. That doesn’t indicate that they’re a good team. I don’t think you can tell anything from CCWM in VEX that you can’t tell from a team’s win/loss record.
I definitely don’t think either is useful if you’re trying to create an unbiased estimator of team performance.
What would you include in a comprehensive ranking then? Some things we’ve considered are:
*]Average match score
*]Average match difficulty
*]Average Margin (how much you win by)
I’m interested to hear what you would factor into your ranking for overall strength.
I think it’s a very difficult problem because the only information that you know for sure is of any value is match wins and anything you derive from them, such as match difficulty. That makes it difficult to improve on the WP/TRSP system (which is pretty clumsy) without introducing biases towards particular strategies.
I feel that things that happen within the match, or things like awards that happen off the field, aren’t going to be equally desirable to different alliance captains. A team that plays defensively and wants to choose an offensive partner would be interested in things like OPR and driver skills scores. A team with a low-scoring autonomous that is designed primarily to get them field position might want a high-scoring autonomous partner - so they would be interested in things like autonomous bonus win %, programming skills and the Think Award.
I don’t think an “overall strength ranking” based on the attributes that 5327 values would be a bad thing to have, but given the extent to which different teams have different priorities I don’t know how many people would see it as authoritative.
One thing that might be useful to have is a ranking based on pre-worlds season performance. We did this out of curiosity last year just before Worlds, using the Wilson score confidence method discussed here (where “positive ratings” = wins and “negative ratings” = losses). This has the advantage of rating teams who consistently achieve good results above those who got good results over a small number of games - for example a team with a season record of 40-10 would score a lot better than one with a season record of 4-1. You can adjust the importance of reliability vs. win % by changing the size of the confidence interval you use.
One problem is that a lot of the most interesting statistics which tournament manager outputs, such as where points were scored, aren’t downloadable from RobotEvents. They were during Gateway, but for Sack Attack and Toss Up the format of the .csv file you can download from RobotEvents has had the same columns as the Gateway one and none of the Sack Attack or Toss Up score breakdown data.
Actually better to include include them, and whatever other generally accepted metrics, and let teams decide what stats/rankings they feel is important. Each metric should have a page with a good discussion of that metric and what is can and cannot measure.
Yeah, we kind of just threw that whole match schedule stuff in last minute in time for Worlds… But we definitely have major interface improvements planned, too!
I agree! Still, it’s a lot to implement. Ideally, we’ll have at least a few metrics set up before most competitions start, but at the very least we should have our raw data from current competitions set up by the end of summer.